{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 学习算法的实现\n",
    "\n",
    "关于神经网络学习的基础知识，前面已经全部介绍完了。“损失函数”、“mini-batch”、“梯度”、“梯度下降法”等关键词已经陆续登场。<br>\n",
    "我们来确认一下神经网络的学习步骤，神经网络的学习步骤如下所示。<br>\n",
    "\n",
    "**前提**<br>\n",
    "神经网络存在合适的权重和偏置，调整权重和偏置以便拟合训练数据的过程称为“学习”。神经网络的学习分为下面4个步骤。\n",
    "\n",
    "**步骤1(mini-batch)**<br>\n",
    "从训练数据中随机选出一部分数据，这部分数据称为 mini-batch。我们的目标是减小 mini-batch的 损失函数的值。<br>\n",
    "\n",
    "**步骤2(计算梯度)**<br>\n",
    "为了减小 mini-batch 的损失函数的值，需要求出各个权重参数的梯度。梯度表示损失函数的值减小最多的方向。<br>\n",
    "\n",
    "**步骤3(更新参数)**<br>\n",
    "将权重参数沿梯度方向进行微小更新。\n",
    "\n",
    "**步骤4(重复)**<br>\n",
    "重复步骤1、步骤、步骤3。\n",
    "\n",
    "\n",
    "神经网络的学习按照上面4个步骤进行。这个方法通过梯度下降法更新参数，不过因为这里使用的数据是随机选择的 mini batch 数据，所以又称**随机梯度下降法**(stochastic gradient descent)。“随机”指的是“随机选择的”意思，因此，随机梯度下降法是“对随机选择的数据进行的梯度下降法”。深度学习的很多框架中，随机梯度下降法一般由一个名为 **SGD** 的函数来实现。SGD 来源于随机梯度下降法的英文名称的首字母。\n",
    "\n",
    "下面，我们来实现手写数字识别的神经网络。这里以2层神经网络(隐藏层为1层的网络)为对象，使用 MNIST 数据集进行学习。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2层神经网络的类\n",
    "首先，我们将这个2层神经网络实现为一个名为 TwoLayerNet 的类，实现过程如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys, os\n",
    "sys.path.append(os.pardir)  # 为了导入父目录的文件而进行的设定\n",
    "from common.functions import *\n",
    "from common.gradient import numerical_gradient\n",
    "\n",
    "\n",
    "class TwoLayerNet:\n",
    "\n",
    "    def __init__(self, input_size, hidden_size, output_size, weight_init_std=0.01):\n",
    "        # 初始化权重\n",
    "        self.params = {}\n",
    "        self.params['W1'] = weight_init_std * np.random.randn(input_size, hidden_size)\n",
    "        self.params['b1'] = np.zeros(hidden_size)\n",
    "        self.params['W2'] = weight_init_std * np.random.randn(hidden_size, output_size)\n",
    "        self.params['b2'] = np.zeros(output_size)\n",
    "\n",
    "    def predict(self, x):\n",
    "        W1, W2 = self.params['W1'], self.params['W2']\n",
    "        b1, b2 = self.params['b1'], self.params['b2']\n",
    "    \n",
    "        a1 = np.dot(x, W1) + b1\n",
    "        z1 = sigmoid(a1)\n",
    "        a2 = np.dot(z1, W2) + b2\n",
    "        y = softmax(a2)\n",
    "        \n",
    "        return y\n",
    "        \n",
    "    # x:输入数据, t:监督数据\n",
    "    def loss(self, x, t):\n",
    "        y = self.predict(x)\n",
    "        \n",
    "        return cross_entropy_error(y, t)\n",
    "    \n",
    "    def accuracy(self, x, t):\n",
    "        y = self.predict(x)\n",
    "        y = np.argmax(y, axis=1)\n",
    "        t = np.argmax(t, axis=1)\n",
    "        \n",
    "        accuracy = np.sum(y == t) / float(x.shape[0])\n",
    "        return accuracy\n",
    "        \n",
    "    # x:输入数据, t:监督数据\n",
    "    def numerical_gradient(self, x, t):\n",
    "        loss_W = lambda W: self.loss(x, t)\n",
    "        \n",
    "        grads = {}\n",
    "        grads['W1'] = numerical_gradient(loss_W, self.params['W1'])\n",
    "        grads['b1'] = numerical_gradient(loss_W, self.params['b1'])\n",
    "        grads['W2'] = numerical_gradient(loss_W, self.params['W2'])\n",
    "        grads['b2'] = numerical_gradient(loss_W, self.params['b2'])\n",
    "        \n",
    "        return grads\n",
    "        \n",
    "    def gradient(self, x, t):\n",
    "        W1, W2 = self.params['W1'], self.params['W2']\n",
    "        b1, b2 = self.params['b1'], self.params['b2']\n",
    "        grads = {}\n",
    "        \n",
    "        batch_num = x.shape[0]\n",
    "        \n",
    "        # forward\n",
    "        a1 = np.dot(x, W1) + b1\n",
    "        z1 = sigmoid(a1)\n",
    "        a2 = np.dot(z1, W2) + b2\n",
    "        y = softmax(a2)\n",
    "        \n",
    "        # backward\n",
    "        dy = (y - t) / batch_num\n",
    "        grads['W2'] = np.dot(z1.T, dy)\n",
    "        grads['b2'] = np.sum(dy, axis=0)\n",
    "        \n",
    "        da1 = np.dot(dy, W2.T)\n",
    "        dz1 = sigmoid_grad(a1) * da1\n",
    "        grads['W1'] = np.dot(x.T, dz1)\n",
    "        grads['b1'] = np.sum(dz1, axis=0)\n",
    "\n",
    "        return grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们把这个类中用到的变量和方法整理一下。\n",
    "\n",
    "$$TwoLayerNet 类中使用的变量$$\n",
    "\n",
    "\n",
    "| 变量 |说明  |\n",
    "|:---|:----|\n",
    "| params | 保存到神经网络的参数的字典型变量(示例变量)。<br>params['W1']是第1层的权重，params['b1']是第2层的偏置。<br>params['W2']是第2层的权重，params['b2']是第2层的偏置。\n",
    "| grads | 保存梯度的字典型变量(numerical_gradient()方法的返回值)。<br>grads['W1']是第1层权重的梯度，grads['b1']是第1层偏置的梯度。<br>grads['W2']是第2层权重的梯度，grads['b2']是第2层偏置的梯度。\n",
    "\n",
    "\n",
    "\n",
    "$$TwoLayerNet 类的方法$$\n",
    "\n",
    "\n",
    "| 方法 |说明  |\n",
    "|:---|:----|\n",
    "| \\__init\\__(self, input_size, hidden_size, output_size) | 进行初始化。<br> 参数从头开始依次表示输入层的神经元数、隐藏层的神经元数、输出层的神经元数。\n",
    "| predict(self, x) | 进行识别(推理)。<br> 参数x是图像数据。\n",
    "| loss(self, x, t) | 计算损失函数的值。<br>参数x是图像数据，t是正确解标签（后面3个方法的参数也一样）。\n",
    "| accuracyy(self, x, t) | 计算识别精度\n",
    "| numerical_gradient(self, x, t) | 计算权重参数的梯度\n",
    "| gradient(self, x, t) | 计算权重参数的梯度。<br>numerical_gradient()的高速版，将在下一章实现。\n",
    "\n",
    "\n",
    "\n",
    "TwoLayerNet 类有 params 和 grads 两个字典型实例变量。params 变量中保存了权重参数，比如 params['W1'] 以 NumPy 数组的形式保存了第1层的权重参数。此外，第1层的偏置可以通过 params['b1'] 进行访问。这里来看一个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10,)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = TwoLayerNet(input_size=784, hidden_size=100, output_size=10)\n",
    "net.params['W1'].shape # (784,100)\n",
    "net.params['b1'].shape # (100,)\n",
    "net.params['W2'].shape # (100,10)\n",
    "net.params['b2'].shape # (10,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如上所示，params变量中保存了该神经网络所需的全部参数。并且，params 变量中保存的权重参数会用在推理处理(前向处理)中。<br>\n",
    "推理处理的实现如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.random.rand(100, 784) # 伪输入数据（100笔）\n",
    "y = net.predict(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外，与 params 变量对应，grads 变量中保存了各个参数的梯度。如下所示，使用 numerical_gradient() 方法计算梯度后，梯度的信息将保存在 grads 变量中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10,)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.random.rand(100, 784) # 伪输入数据（100笔）\n",
    "t = np.random.rand(100, 10)  # 伪正确解标签(100笔)\n",
    "\n",
    "grads = net.numerical_gradient(x, t) # 计算梯度\n",
    "\n",
    "grads['W1'].shape # (784,100)\n",
    "grads['b1'].shape # (100,)\n",
    "grads['W2'].shape # (100,10)\n",
    "grads['b2'].shape # (10,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着，我们看一下 TwoLayerNet 的方法的实现。首先是 \\__init\\__(self, input_size, hidden_size, output_size)方法，它是类的初始化方法（生成 TwoLayerNet 示例时被调用的方法）。从第1个参数开始、依次表示输入层的神经元数、隐藏层的神经元数、输出层的神经元数。另外，因为进行手写数字识别时，输入图像的大小是784(28x28)，输出为10个类别，所以指定参数 input_size=784、output_size=10，将隐藏层的个数 hidden_size 设置为一个合适的值即可。\n",
    "\n",
    "此外，这个初始化方法会对权重参数进行初始化。如何设置权重参数的初始值这个问题是关系到神经网络能否成功学习的重要问题。后面会详细讨论权重参数的初始化，这里只需要知道，**权重使用符合高斯分布的随机数进行初始化，偏置使用0进行初始化**。<br>\n",
    "predict(self, x) 和 accuracy(self, x,t)的实现和上一章的神经网络的推理处理基本一样。<br>\n",
    "loss(self, x, t) 是计算函数损失值的方法。这个方法会基于 predict() 的结果和正确解标签，计算交叉熵误差。<br>\n",
    "numerical_gradient(self, x, t) 方法会计算各个参数的梯度。根据数值微分，计算各个参数相对于损失函数的梯度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### mini-batch 的实现\n",
    "\n",
    "神经网络的学习的实现使用的是前面介绍过的 mini-batch xuexi 。所谓 mini-batch 学习，就是从训练数据中随机选择一部分数据(成为 mini-batch)，再以这些 mini-batch 为对象，使用梯度法更新参数的过程。下面，我们以 TwoLayerNet 类为对象，使用 MNIST 数据集进行学习。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "2.301028394024926\n",
      "1\n",
      "2.2903224128948465\n",
      "2\n",
      "2.2809646806511665\n",
      "3\n",
      "2.2884177868963906\n",
      "4\n",
      "2.299284214734375\n",
      "5\n",
      "2.287838394450531\n",
      "6\n",
      "2.296233959560264\n",
      "7\n",
      "2.297510114141158\n",
      "8\n",
      "2.290980195072725\n",
      "9\n",
      "2.301388310868444\n",
      "10\n",
      "2.294136017083701\n",
      "11\n",
      "2.291691967858927\n",
      "12\n",
      "2.2913137903853174\n",
      "13\n",
      "2.2978196497987935\n",
      "14\n",
      "2.293203532807744\n",
      "15\n",
      "2.2866375955238167\n",
      "16\n",
      "2.2843412293283625\n",
      "17\n",
      "2.269499349919448\n",
      "18\n",
      "2.296239287589886\n",
      "19\n",
      "2.287970389757371\n",
      "20\n",
      "2.2900892831506345\n",
      "21\n",
      "2.2821764935439783\n",
      "22\n",
      "2.2945760781962776\n",
      "23\n",
      "2.286840339262744\n",
      "24\n",
      "2.2866166411249393\n",
      "25\n",
      "2.280128188630702\n",
      "26\n",
      "2.2847117576998177\n",
      "27\n",
      "2.293226526250327\n",
      "28\n",
      "2.2782659768993976\n",
      "29\n",
      "2.3042279650131103\n",
      "30\n",
      "2.293636693483892\n",
      "31\n",
      "2.288824428761272\n",
      "32\n",
      "2.2745768609842587\n",
      "33\n",
      "2.2939163944738308\n",
      "34\n",
      "2.282050720899493\n",
      "35\n",
      "2.2899233202817753\n",
      "36\n",
      "2.2644081363886652\n",
      "37\n",
      "2.273063568845338\n",
      "38\n",
      "2.289885379145292\n",
      "39\n",
      "2.278044416601021\n",
      "40\n",
      "2.2870143466442205\n",
      "41\n",
      "2.3066229607898006\n",
      "42\n",
      "2.280564265014839\n",
      "43\n",
      "2.2532826347733055\n",
      "44\n",
      "2.2916445878156644\n",
      "45\n",
      "2.27451929658718\n",
      "46\n",
      "2.2812088400795223\n",
      "47\n",
      "2.2879965666569593\n",
      "48\n",
      "2.2765227187476356\n",
      "49\n",
      "2.284888898871421\n",
      "50\n",
      "2.2935591403832434\n",
      "51\n",
      "2.2767007793250613\n",
      "52\n",
      "2.2634106875640283\n",
      "53\n",
      "2.271717678634471\n",
      "54\n",
      "2.280732744145497\n",
      "55\n",
      "2.3063010612266983\n",
      "56\n",
      "2.2940285975325128\n",
      "57\n",
      "2.28286420665564\n",
      "58\n",
      "2.2994618141855243\n",
      "59\n",
      "2.2942187550400983\n",
      "60\n",
      "2.2908171171063048\n",
      "61\n",
      "2.2621728855609438\n",
      "62\n",
      "2.281719555202025\n",
      "63\n",
      "2.288519483322783\n",
      "64\n",
      "2.2819510880188343\n",
      "65\n",
      "2.27516294354143\n",
      "66\n",
      "2.266770310184049\n",
      "67\n",
      "2.27741258318551\n",
      "68\n",
      "2.275073290146258\n",
      "69\n",
      "2.290817532173699\n",
      "70\n",
      "2.2937689587308903\n",
      "71\n",
      "2.295205600245503\n",
      "72\n",
      "2.2899487869584774\n",
      "73\n",
      "2.292681800049381\n",
      "74\n",
      "2.278133640247843\n",
      "75\n",
      "2.295688870895661\n",
      "76\n",
      "2.2838412022942665\n",
      "77\n",
      "2.2844349343841714\n",
      "78\n",
      "2.261733566677796\n",
      "79\n",
      "2.2655707153784674\n",
      "80\n",
      "2.2932639767523804\n",
      "81\n",
      "2.2911424969640968\n",
      "82\n",
      "2.27661904153836\n",
      "83\n",
      "2.2673805904747852\n",
      "84\n",
      "2.2881730315853575\n",
      "85\n",
      "2.2717926356534686\n",
      "86\n",
      "2.2830223385484056\n",
      "87\n",
      "2.287541216231614\n",
      "88\n",
      "2.267205540199579\n",
      "89\n",
      "2.2781920292716413\n",
      "90\n",
      "2.267229579697796\n",
      "91\n",
      "2.267184411922525\n",
      "92\n",
      "2.2776154975256278\n",
      "93\n",
      "2.280436613975786\n",
      "94\n",
      "2.2737217226979705\n",
      "95\n",
      "2.2879606047225916\n",
      "96\n",
      "2.2727558275546453\n",
      "97\n",
      "2.2770565902746953\n",
      "98\n",
      "2.2730256195178216\n",
      "99\n",
      "2.271199940438595\n",
      "100\n",
      "2.277417492450792\n",
      "101\n",
      "2.2714161162494713\n",
      "102\n",
      "2.2911536448833374\n",
      "103\n",
      "2.2856268412331575\n",
      "104\n",
      "2.2812960063271404\n",
      "105\n",
      "2.2577192490637454\n",
      "106\n",
      "2.273228992376064\n",
      "107\n",
      "2.277467834498756\n",
      "108\n",
      "2.2732254639161926\n",
      "109\n",
      "2.28084014833343\n",
      "110\n",
      "2.261857977206499\n",
      "111\n",
      "2.2708275356032614\n",
      "112\n",
      "2.2652382950074537\n",
      "113\n",
      "2.2700209575893067\n",
      "114\n",
      "2.2732182248664077\n",
      "115\n",
      "2.2600408756988823\n",
      "116\n",
      "2.2708628725040545\n",
      "117\n",
      "2.279379022177462\n",
      "118\n",
      "2.260821825703223\n",
      "119\n",
      "2.2644456743158257\n",
      "120\n",
      "2.283217847557467\n",
      "121\n",
      "2.2606647668595605\n",
      "122\n",
      "2.2628673643312687\n",
      "123\n",
      "2.267799490834218\n",
      "124\n",
      "2.2773285182426166\n",
      "125\n",
      "2.2388898898392764\n",
      "126\n",
      "2.2760427983612495\n",
      "127\n",
      "2.25096192473043\n",
      "128\n",
      "2.273192260353345\n",
      "129\n",
      "2.2588916362970557\n",
      "130\n",
      "2.233270900349797\n",
      "131\n",
      "2.2621944026681864\n",
      "132\n",
      "2.2575710119264385\n",
      "133\n",
      "2.251828463388778\n",
      "134\n",
      "2.2467461039499237\n",
      "135\n",
      "2.258513361836435\n",
      "136\n",
      "2.265685249435983\n",
      "137\n",
      "2.253148448311314\n",
      "138\n",
      "2.2299363120328017\n",
      "139\n",
      "2.228989665876026\n",
      "140\n",
      "2.2453376044056137\n",
      "141\n",
      "2.2484283724234917\n",
      "142\n",
      "2.224490738446411\n",
      "143\n",
      "2.2511153528723047\n",
      "144\n",
      "2.2486966976064515\n",
      "145\n",
      "2.22941995001112\n",
      "146\n",
      "2.250512961315265\n",
      "147\n",
      "2.2406595168368906\n",
      "148\n",
      "2.2443967028854805\n",
      "149\n",
      "2.2485101505234173\n",
      "150\n",
      "2.245849500694558\n",
      "151\n",
      "2.236259148331965\n",
      "152\n",
      "2.2307977095266573\n",
      "153\n",
      "2.231290439549239\n",
      "154\n",
      "2.232529245454183\n",
      "155\n",
      "2.2365916285438163\n",
      "156\n",
      "2.234850588364191\n",
      "157\n",
      "2.2309550586086577\n",
      "158\n",
      "2.215346790717187\n",
      "159\n",
      "2.2201693207332216\n",
      "160\n",
      "2.2194499068113815\n",
      "161\n",
      "2.220027685570546\n",
      "162\n",
      "2.2261273722618196\n",
      "163\n",
      "2.2188840242967505\n",
      "164\n",
      "2.2230127763210863\n",
      "165\n",
      "2.2214290191187196\n",
      "166\n",
      "2.1995748217281634\n",
      "167\n",
      "2.209432735892524\n",
      "168\n",
      "2.2084023085443723\n",
      "169\n",
      "2.2027207821070847\n",
      "170\n",
      "2.216326279226732\n",
      "171\n",
      "2.212143890241052\n",
      "172\n",
      "2.1983016759173615\n",
      "173\n",
      "2.1910825461470242\n",
      "174\n",
      "2.1805986403055444\n",
      "175\n",
      "2.208128475088571\n",
      "176\n",
      "2.1935412453602594\n",
      "177\n",
      "2.2189356883889384\n",
      "178\n",
      "2.1732772359861623\n",
      "179\n",
      "2.181621568240201\n",
      "180\n",
      "2.170630165472359\n",
      "181\n",
      "2.17804535516186\n",
      "182\n",
      "2.161756744632081\n",
      "183\n",
      "2.1494207442394324\n",
      "184\n",
      "2.1564500305256895\n",
      "185\n",
      "2.148289387994236\n",
      "186\n",
      "2.197178899089337\n",
      "187\n",
      "2.1731471107401363\n",
      "188\n",
      "2.14052402541898\n",
      "189\n",
      "2.1921502096361185\n",
      "190\n",
      "2.1580263062723573\n",
      "191\n",
      "2.1319321001521674\n",
      "192\n",
      "2.166147511963745\n",
      "193\n",
      "2.1643901268834473\n",
      "194\n",
      "2.128036484639974\n",
      "195\n",
      "2.1591577373609634\n",
      "196\n",
      "2.1212992762090614\n",
      "197\n",
      "2.161060661034656\n",
      "198\n",
      "2.131402205679259\n",
      "199\n",
      "2.1181436078885243\n",
      "200\n",
      "2.1484008619052655\n",
      "201\n",
      "2.114460941856606\n",
      "202\n",
      "2.128388896371812\n",
      "203\n",
      "2.1454297981326484\n",
      "204\n",
      "2.100982149808286\n",
      "205\n",
      "2.0898918750235342\n",
      "206\n",
      "2.1461684805885324\n",
      "207\n",
      "2.120098489936294\n",
      "208\n",
      "2.102709332118529\n",
      "209\n",
      "2.1186681993518888\n",
      "210\n",
      "2.099614063169341\n",
      "211\n",
      "2.105038283590131\n",
      "212\n",
      "2.1008257977946867\n",
      "213\n",
      "2.1284675513539146\n",
      "214\n",
      "2.095823667164679\n",
      "215\n",
      "2.0813015297552506\n",
      "216\n",
      "2.059837012785963\n",
      "217\n",
      "2.098302539439531\n",
      "218\n",
      "2.0673570255900255\n",
      "219\n",
      "2.035817194355214\n",
      "220\n",
      "2.0341467247705727\n",
      "221\n",
      "2.0631077081680576\n",
      "222\n",
      "2.0519476714699207\n",
      "223\n",
      "2.0733662190899524\n",
      "224\n",
      "2.057082846981657\n",
      "225\n",
      "2.0354962993209433\n",
      "226\n",
      "2.081147288659261\n",
      "227\n",
      "2.0604091920922794\n",
      "228\n",
      "2.0856656574550674\n",
      "229\n",
      "2.028804128767087\n",
      "230\n",
      "2.042706450279057\n",
      "231\n",
      "2.05961496495565\n",
      "232\n",
      "1.9700155729824602\n",
      "233\n",
      "2.0432645147208364\n",
      "234\n",
      "2.0261805570815286\n",
      "235\n",
      "2.018038728218572\n",
      "236\n",
      "1.9867416821459536\n",
      "237\n",
      "1.9900088492384505\n",
      "238\n",
      "1.978595043013816\n",
      "239\n",
      "2.018054910250572\n",
      "240\n",
      "2.0022093800160894\n",
      "241\n",
      "1.9556689198161934\n",
      "242\n",
      "1.9209105323742737\n",
      "243\n",
      "1.9738341935869925\n",
      "244\n",
      "1.9784630983915843\n",
      "245\n",
      "1.9608665158552336\n",
      "246\n",
      "1.9177642333329317\n",
      "247\n",
      "1.9272743060968807\n",
      "248\n",
      "1.9948012510775956\n",
      "249\n",
      "1.970844889706656\n",
      "250\n",
      "1.9931079444289859\n",
      "251\n",
      "1.9020147740598348\n",
      "252\n",
      "1.934518168977708\n",
      "253\n",
      "1.9395085936619674\n",
      "254\n",
      "1.915061312641783\n",
      "255\n",
      "1.9201724265668212\n",
      "256\n",
      "1.9631208214538876\n",
      "257\n",
      "1.8870216055520657\n",
      "258\n",
      "1.9226836133382188\n",
      "259\n",
      "1.9006454610573862\n",
      "260\n",
      "1.9033522170075639\n",
      "261\n",
      "1.8673001877641662\n",
      "262\n",
      "1.9111267852594356\n",
      "263\n",
      "1.8651668205239949\n",
      "264\n",
      "1.9009996475872262\n",
      "265\n",
      "1.9514207744684426\n",
      "266\n",
      "1.8371734022898685\n",
      "267\n",
      "1.8177618908330433\n",
      "268\n",
      "1.8120001035413182\n",
      "269\n",
      "1.8170278393974142\n",
      "270\n",
      "1.8425960807404045\n",
      "271\n",
      "1.8666890420031865\n",
      "272\n",
      "1.8001791696590124\n",
      "273\n",
      "1.8418304564897476\n",
      "274\n",
      "1.8000534782940119\n",
      "275\n",
      "1.8405196290717307\n",
      "276\n",
      "1.8680594026250512\n",
      "277\n",
      "1.7978395923817359\n",
      "278\n",
      "1.699178406329131\n",
      "279\n",
      "1.8173981669125376\n",
      "280\n",
      "1.7735862593216885\n",
      "281\n",
      "1.7473913076058423\n",
      "282\n",
      "1.7665416958498465\n",
      "283\n",
      "1.8087329246651422\n",
      "284\n",
      "1.754178210517276\n",
      "285\n",
      "1.707936568554353\n",
      "286\n",
      "1.7786879070245933\n",
      "287\n",
      "1.7434758563050883\n",
      "288\n",
      "1.6874142045907525\n",
      "289\n",
      "1.8013845407535782\n",
      "290\n",
      "1.7466199894805743\n",
      "291\n",
      "1.6739864945810334\n",
      "292\n",
      "1.6637923263076837\n",
      "293\n",
      "1.7470406752711978\n",
      "294\n",
      "1.7332827162531554\n",
      "295\n",
      "1.7986358582175972\n",
      "296\n",
      "1.6914971876330815\n",
      "297\n",
      "1.6938974887199973\n",
      "298\n",
      "1.753410893820247\n",
      "299\n",
      "1.684968762595789\n",
      "300\n",
      "1.6543954775368592\n",
      "301\n",
      "1.5819345507355422\n",
      "302\n",
      "1.7335714981167356\n",
      "303\n",
      "1.5870121656424119\n",
      "304\n",
      "1.6800936810385037\n",
      "305\n",
      "1.7189227437636034\n",
      "306\n",
      "1.6329996700519167\n",
      "307\n",
      "1.6161626722465636\n",
      "308\n",
      "1.6296616832340667\n",
      "309\n",
      "1.6249857864794288\n",
      "310\n",
      "1.6424082710335974\n",
      "311\n",
      "1.6250140833308435\n",
      "312\n",
      "1.6708507438251232\n",
      "313\n",
      "1.6065664531651982\n",
      "314\n",
      "1.6378425931783405\n",
      "315\n",
      "1.638942605365895\n",
      "316\n",
      "1.5843508759564358\n",
      "317\n",
      "1.6026082953917662\n",
      "318\n",
      "1.5905265890228835\n",
      "319\n",
      "1.5878381779868445\n",
      "320\n",
      "1.693202670756665\n",
      "321\n",
      "1.5925500047489356\n",
      "322\n",
      "1.5957423244619486\n",
      "323\n",
      "1.621783414474129\n",
      "324\n",
      "1.5385325832516197\n",
      "325\n",
      "1.5581920086610337\n",
      "326\n",
      "1.5684601424680946\n",
      "327\n",
      "1.5223723990294522\n",
      "328\n",
      "1.6419593398302186\n",
      "329\n",
      "1.5880064361599608\n",
      "330\n",
      "1.488953193436016\n",
      "331\n",
      "1.5018873147363274\n",
      "332\n",
      "1.5401953809393694\n",
      "333\n",
      "1.5114281664227485\n",
      "334\n",
      "1.4629737076134604\n",
      "335\n",
      "1.4486314017783066\n",
      "336\n",
      "1.5188795927262717\n",
      "337\n",
      "1.5328102722571841\n",
      "338\n",
      "1.3625233258260514\n",
      "339\n",
      "1.4677043291919503\n",
      "340\n",
      "1.4805320775918989\n",
      "341\n",
      "1.5560465349114012\n",
      "342\n",
      "1.4677322740175607\n",
      "343\n",
      "1.5134479235065843\n",
      "344\n",
      "1.4740714812422502\n",
      "345\n",
      "1.4467223248299024\n",
      "346\n",
      "1.423633079652268\n",
      "347\n",
      "1.5180162145810874\n",
      "348\n",
      "1.5140670365953486\n",
      "349\n",
      "1.5236826548637474\n",
      "350\n",
      "1.4379139450740501\n",
      "351\n",
      "1.4926829376170323\n",
      "352\n",
      "1.456119127763519\n",
      "353\n",
      "1.3622008048442695\n",
      "354\n",
      "1.472296912009683\n",
      "355\n",
      "1.4631043240233739\n",
      "356\n",
      "1.5015585320661595\n",
      "357\n",
      "1.372069789353366\n",
      "358\n",
      "1.4732109003117788\n",
      "359\n",
      "1.4469547658899402\n",
      "360\n",
      "1.4246207701871072\n",
      "361\n",
      "1.3968448849711763\n",
      "362\n",
      "1.3403427717847145\n",
      "363\n",
      "1.415377703108696\n",
      "364\n",
      "1.3332467115496909\n",
      "365\n",
      "1.459836611354514\n",
      "366\n",
      "1.369818644298613\n",
      "367\n",
      "1.3604045762786752\n",
      "368\n",
      "1.4924251975234935\n",
      "369\n",
      "1.457236484904094\n",
      "370\n",
      "1.4625182872113047\n",
      "371\n",
      "1.4055770511212407\n",
      "372\n",
      "1.386173065338365\n",
      "373\n",
      "1.2919126696007246\n",
      "374\n",
      "1.3770734827523576\n",
      "375\n",
      "1.4131777394379983\n",
      "376\n",
      "1.445952527664743\n",
      "377\n",
      "1.2604020430327298\n",
      "378\n",
      "1.2557361236201086\n",
      "379\n",
      "1.3151167131389838\n",
      "380\n",
      "1.3303092680899702\n",
      "381\n",
      "1.3551626466160596\n",
      "382\n",
      "1.335626723640121\n",
      "383\n",
      "1.2937141222522373\n",
      "384\n",
      "1.3975071936232597\n",
      "385\n",
      "1.3243348359942888\n",
      "386\n",
      "1.309123501468609\n",
      "387\n",
      "1.2635239410648358\n",
      "388\n",
      "1.3204129752708098\n",
      "389\n",
      "1.3797355533716975\n",
      "390\n",
      "1.2112014461436695\n",
      "391\n",
      "1.270983720191074\n",
      "392\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2171795925461353\n",
      "393\n",
      "1.2920780073411762\n",
      "394\n",
      "1.3184617566832508\n",
      "395\n",
      "1.295408597794069\n",
      "396\n",
      "1.2416028089763405\n",
      "397\n",
      "1.2723321928373201\n",
      "398\n",
      "1.2478184527317915\n",
      "399\n",
      "1.294072939208798\n",
      "400\n",
      "1.3793715633460868\n",
      "401\n",
      "1.3131467284662919\n",
      "402\n",
      "1.2677165528270422\n",
      "403\n",
      "1.2810023464724023\n",
      "404\n",
      "1.210153401149337\n",
      "405\n",
      "1.3282185975686696\n",
      "406\n",
      "1.3161424386196954\n",
      "407\n",
      "1.3080596700810267\n",
      "408\n",
      "1.2308396551940795\n",
      "409\n",
      "1.2675457390204117\n",
      "410\n",
      "1.292214807273183\n",
      "411\n",
      "1.32181336194773\n",
      "412\n",
      "1.297092799216741\n",
      "413\n",
      "1.34246870933181\n",
      "414\n",
      "1.2440026263470023\n",
      "415\n",
      "1.0999360963198657\n",
      "416\n",
      "1.251720238694534\n",
      "417\n",
      "1.176053585289263\n",
      "418\n",
      "1.246011726827991\n",
      "419\n",
      "1.2513654126291103\n",
      "420\n",
      "1.2517139333643634\n",
      "421\n",
      "1.214196057025773\n",
      "422\n",
      "1.1703302053293618\n",
      "423\n",
      "1.205111868589013\n",
      "424\n",
      "1.2756981670651117\n",
      "425\n",
      "1.0776172316315071\n",
      "426\n",
      "1.1859566977386122\n",
      "427\n",
      "1.2395440270521578\n",
      "428\n",
      "1.22833880834402\n",
      "429\n",
      "1.1588706226531396\n",
      "430\n",
      "1.172856111797651\n",
      "431\n",
      "1.1646465509256516\n",
      "432\n",
      "1.125548201488257\n",
      "433\n",
      "1.1198346553412168\n",
      "434\n",
      "1.2185877391080213\n",
      "435\n",
      "1.0377111168338715\n",
      "436\n",
      "1.198630539738244\n",
      "437\n",
      "1.1345544029643424\n",
      "438\n",
      "1.1448428939220001\n",
      "439\n",
      "1.0944256652387663\n",
      "440\n",
      "1.1431176525317805\n",
      "441\n",
      "1.107482693487195\n",
      "442\n",
      "1.242628849775993\n",
      "443\n",
      "1.102257950504644\n",
      "444\n",
      "1.1296814176850964\n",
      "445\n",
      "1.1886497317035947\n",
      "446\n",
      "1.1612596764597414\n",
      "447\n",
      "1.0716858289203435\n",
      "448\n",
      "1.1306897096146293\n",
      "449\n",
      "1.1005995423995605\n",
      "450\n",
      "1.098876578675818\n",
      "451\n",
      "1.0270965671210308\n",
      "452\n",
      "1.180376602585855\n",
      "453\n",
      "1.1753206860796457\n",
      "454\n",
      "1.1865723954782914\n",
      "455\n",
      "1.1299995211494704\n",
      "456\n",
      "1.0248068032842763\n",
      "457\n",
      "1.0454063528696496\n",
      "458\n",
      "1.090125387769978\n",
      "459\n",
      "1.2729283787112025\n",
      "460\n",
      "0.9699564132524237\n",
      "461\n",
      "1.1140387941155616\n",
      "462\n",
      "1.0998370596355254\n",
      "463\n",
      "1.2325838902255364\n",
      "464\n",
      "0.9952982454600492\n",
      "465\n",
      "1.1125915456285091\n",
      "466\n",
      "1.01904486800739\n",
      "467\n",
      "1.056465285715996\n",
      "468\n",
      "1.0170604861099861\n",
      "469\n",
      "1.0998736349451321\n",
      "470\n",
      "1.0778945379857054\n",
      "471\n",
      "1.184241986976936\n",
      "472\n",
      "1.0805256104582446\n",
      "473\n",
      "0.9930745892224544\n",
      "474\n",
      "1.1501885463378385\n",
      "475\n",
      "1.0164475877780366\n",
      "476\n",
      "1.1172241087827186\n",
      "477\n",
      "0.9453956878853756\n",
      "478\n",
      "0.9638382788237791\n",
      "479\n",
      "1.1065879346791738\n",
      "480\n",
      "1.0775762503133703\n",
      "481\n",
      "1.090255290885129\n",
      "482\n",
      "0.9496612042515069\n",
      "483\n",
      "1.109586445070655\n",
      "484\n",
      "1.0391592634013642\n",
      "485\n",
      "1.1212230895442001\n",
      "486\n",
      "1.0215024491846452\n",
      "487\n",
      "1.1951538776770567\n",
      "488\n",
      "1.0004190992945998\n",
      "489\n",
      "1.0297556629945384\n",
      "490\n",
      "1.0771959566959197\n",
      "491\n",
      "1.145316680338371\n",
      "492\n",
      "1.1173405560845973\n",
      "493\n",
      "1.037257086249722\n",
      "494\n",
      "0.9364119289263308\n",
      "495\n",
      "1.0327034130031345\n",
      "496\n",
      "1.0385060721490087\n",
      "497\n",
      "1.046311325931846\n",
      "498\n",
      "0.9498558619682684\n",
      "499\n",
      "0.9423017714347627\n",
      "500\n",
      "1.0263184703717607\n",
      "501\n",
      "0.9740635241241814\n",
      "502\n",
      "0.9812904529366083\n",
      "503\n",
      "0.9602615769088952\n",
      "504\n",
      "0.9792464976878358\n",
      "505\n",
      "0.9668840429482153\n",
      "506\n",
      "0.9921964944989103\n",
      "507\n",
      "0.9378700317872126\n",
      "508\n",
      "1.0394157067502885\n",
      "509\n",
      "0.9615888441440015\n",
      "510\n",
      "0.9722252456549151\n",
      "511\n",
      "1.0430151131353183\n",
      "512\n",
      "0.9452608807427761\n",
      "513\n",
      "0.9245301573527217\n",
      "514\n",
      "0.981761360551137\n",
      "515\n",
      "0.9807038339531975\n",
      "516\n",
      "1.0220910156428396\n",
      "517\n",
      "0.9410650158478417\n",
      "518\n",
      "1.0051421832765093\n",
      "519\n",
      "1.0100985660827115\n",
      "520\n",
      "0.9376846259565901\n",
      "521\n",
      "0.9283528288913167\n",
      "522\n",
      "0.977962063398214\n",
      "523\n",
      "0.9678396913690976\n",
      "524\n",
      "0.8926229972913394\n",
      "525\n",
      "0.8451915741434546\n",
      "526\n",
      "0.983307109809285\n",
      "527\n",
      "0.9495902210846713\n",
      "528\n",
      "0.9087018510581235\n",
      "529\n",
      "0.9563565964081472\n",
      "530\n",
      "0.9441552390991734\n",
      "531\n",
      "0.9182820302261078\n",
      "532\n",
      "0.9705536812456194\n",
      "533\n",
      "0.9174127338549454\n",
      "534\n",
      "0.8714838144979771\n",
      "535\n",
      "1.0597103377407944\n",
      "536\n",
      "0.9487956995259025\n",
      "537\n",
      "0.9749380816667375\n",
      "538\n",
      "0.9312621680778768\n",
      "539\n",
      "0.98971372343469\n",
      "540\n",
      "0.8692005847005411\n",
      "541\n",
      "1.0704949249955025\n",
      "542\n",
      "0.9339903800455633\n",
      "543\n",
      "0.9200994486635105\n",
      "544\n",
      "0.9162515966563967\n",
      "545\n",
      "0.9065839925435395\n",
      "546\n",
      "0.9632590862956032\n",
      "547\n",
      "0.9194206236539533\n",
      "548\n",
      "0.9284383911023565\n",
      "549\n",
      "0.8966269153063255\n",
      "550\n",
      "1.0365826355820722\n",
      "551\n",
      "0.9081622522045514\n",
      "552\n",
      "0.8554867034325077\n",
      "553\n",
      "0.9682564748118244\n",
      "554\n",
      "0.9237034214870854\n",
      "555\n",
      "0.8940147752666217\n",
      "556\n",
      "0.9186880918544921\n",
      "557\n",
      "1.0185767956933487\n",
      "558\n",
      "0.8352342283840549\n",
      "559\n",
      "0.9109872189846603\n",
      "560\n",
      "0.9595171852071186\n",
      "561\n",
      "0.9945860507734834\n",
      "562\n",
      "0.9325625569907607\n",
      "563\n",
      "0.8300490133214304\n",
      "564\n",
      "0.7643686812900132\n",
      "565\n",
      "0.9493792432076117\n",
      "566\n",
      "0.7552776258645469\n",
      "567\n",
      "1.0014290332372577\n",
      "568\n",
      "1.0068841198426965\n",
      "569\n",
      "0.8652734748147668\n",
      "570\n",
      "0.9217470040701816\n",
      "571\n",
      "0.8611432330908089\n",
      "572\n",
      "0.9793401873202359\n",
      "573\n",
      "0.8485137681226698\n",
      "574\n",
      "0.8085679751117958\n",
      "575\n",
      "0.8691867234085473\n",
      "576\n",
      "0.9216174568776502\n",
      "577\n",
      "0.8815672890001678\n",
      "578\n",
      "0.7985901389481336\n",
      "579\n",
      "0.9156324995570339\n",
      "580\n",
      "1.0034735562005135\n",
      "581\n",
      "0.8746889730902159\n",
      "582\n",
      "0.8055385897005688\n",
      "583\n",
      "0.972516642576263\n",
      "584\n",
      "0.8142967388757526\n",
      "585\n",
      "0.901586498998654\n",
      "586\n",
      "0.8152672989588752\n",
      "587\n",
      "0.8553808369080599\n",
      "588\n",
      "0.8715117566969213\n",
      "589\n",
      "0.8490525545841163\n",
      "590\n",
      "0.8497312844749573\n",
      "591\n",
      "0.8359501879878006\n",
      "592\n",
      "0.8409245776901476\n",
      "593\n",
      "0.8229781536390227\n",
      "594\n",
      "0.7976932527500747\n",
      "595\n",
      "0.750707798171755\n",
      "596\n",
      "0.9257556001413998\n",
      "597\n",
      "0.7940050132091935\n",
      "598\n",
      "0.8938187317461012\n",
      "599\n",
      "0.8405918131302257\n",
      "600\n",
      "0.8954077377997274\n",
      "601\n",
      "0.7380375732774834\n",
      "602\n",
      "0.821148967082884\n",
      "603\n",
      "0.8844093971964131\n",
      "604\n",
      "0.7788880831103587\n",
      "605\n",
      "0.8349163143791869\n",
      "606\n",
      "0.8665366498053152\n",
      "607\n",
      "0.7590165191162701\n",
      "608\n",
      "0.9017989540207086\n",
      "609\n",
      "0.8092330965882282\n",
      "610\n",
      "0.8248103435688656\n",
      "611\n",
      "0.7867750859528264\n",
      "612\n",
      "0.9146578509438335\n",
      "613\n",
      "0.7888031866557903\n",
      "614\n",
      "0.7909700341019048\n",
      "615\n",
      "0.7935616591039019\n",
      "616\n",
      "0.8145383858285691\n",
      "617\n",
      "1.0131687931433653\n",
      "618\n",
      "0.8532917050618629\n",
      "619\n",
      "0.8126369741101016\n",
      "620\n",
      "0.820329649667979\n",
      "621\n",
      "0.8938139649438308\n",
      "622\n",
      "0.7958534764191674\n",
      "623\n",
      "0.8609726217654883\n",
      "624\n",
      "0.9528264677390773\n",
      "625\n",
      "0.7231126359448051\n",
      "626\n",
      "0.9099520630285302\n",
      "627\n",
      "0.7500896432095421\n",
      "628\n",
      "0.8236012104643052\n",
      "629\n",
      "0.8016387714630056\n",
      "630\n",
      "0.87088828180841\n",
      "631\n",
      "0.732822882331948\n",
      "632\n",
      "0.6444246013118925\n",
      "633\n",
      "0.9615550719119379\n",
      "634\n",
      "0.7255391125303479\n",
      "635\n",
      "0.8891250674158689\n",
      "636\n",
      "0.8366768853966058\n",
      "637\n",
      "0.7690391595961572\n",
      "638\n",
      "0.8713889670290343\n",
      "639\n",
      "0.7614509864733583\n",
      "640\n",
      "0.8009783597554587\n",
      "641\n",
      "0.7995028639117375\n",
      "642\n",
      "0.9121888047627681\n",
      "643\n",
      "0.7854146270395479\n",
      "644\n",
      "0.8558239424157564\n",
      "645\n",
      "0.7686155966044982\n",
      "646\n",
      "0.767251071013786\n",
      "647\n",
      "0.7637113473342555\n",
      "648\n",
      "0.8084544879108501\n",
      "649\n",
      "0.7237835227134015\n",
      "650\n",
      "0.7655945841751302\n",
      "651\n",
      "0.825309648651041\n",
      "652\n",
      "0.7298611579664873\n",
      "653\n",
      "0.7804502510385071\n",
      "654\n",
      "0.7782504434875542\n",
      "655\n",
      "0.8529932869251304\n",
      "656\n",
      "0.7993571102436293\n",
      "657\n",
      "0.8117400000001435\n",
      "658\n",
      "0.7568903043276646\n",
      "659\n",
      "0.7898101907868577\n",
      "660\n",
      "0.8291947508969308\n",
      "661\n",
      "0.772646000196999\n",
      "662\n",
      "0.8189063619681075\n",
      "663\n",
      "0.8841962271337342\n",
      "664\n",
      "0.5649307471438931\n",
      "665\n",
      "0.7507092356095697\n",
      "666\n",
      "0.6862669755517655\n",
      "667\n",
      "0.7825754654078046\n",
      "668\n",
      "0.79477736633635\n",
      "669\n",
      "0.7859156113972907\n",
      "670\n",
      "0.7349415569147113\n",
      "671\n",
      "0.6774145266931758\n",
      "672\n",
      "0.7814694316354408\n",
      "673\n",
      "0.7048444702401032\n",
      "674\n",
      "0.6925115161333014\n",
      "675\n",
      "0.7306078765918815\n",
      "676\n",
      "0.7443744276315685\n",
      "677\n",
      "0.7024894335719724\n",
      "678\n",
      "0.7536093294474489\n",
      "679\n",
      "0.8060146361575068\n",
      "680\n",
      "0.7350840136658447\n",
      "681\n",
      "0.6171677544240307\n",
      "682\n",
      "0.639033553373196\n",
      "683\n",
      "0.7357636498078322\n",
      "684\n",
      "0.7983370535770439\n",
      "685\n",
      "0.6735602831609413\n",
      "686\n",
      "0.7515249240402129\n",
      "687\n",
      "0.8462855488944099\n",
      "688\n",
      "0.6981905281408903\n",
      "689\n",
      "0.6048587884372063\n",
      "690\n",
      "0.853177993156181\n",
      "691\n",
      "0.817744346016423\n",
      "692\n",
      "0.7605067599541393\n",
      "693\n",
      "0.671933893803201\n",
      "694\n",
      "0.6873051663515446\n",
      "695\n",
      "0.7049799936961917\n",
      "696\n",
      "0.7353750834489067\n",
      "697\n",
      "0.7468387871879767\n",
      "698\n",
      "0.8185027906609124\n",
      "699\n",
      "0.7995093668894664\n",
      "700\n",
      "0.7470166650608324\n",
      "701\n",
      "0.6332676979534876\n",
      "702\n",
      "0.8088609563538816\n",
      "703\n",
      "0.8573045576532493\n",
      "704\n",
      "0.7979982857034613\n",
      "705\n",
      "0.7324490301334201\n",
      "706\n",
      "0.6450819674295326\n",
      "707\n",
      "0.64838288582808\n",
      "708\n",
      "0.8165589706753568\n",
      "709\n",
      "0.6478605114872711\n",
      "710\n",
      "0.6516472814809207\n",
      "711\n",
      "0.6699602096194621\n",
      "712\n",
      "0.7215437422953354\n",
      "713\n",
      "0.6947449525315939\n",
      "714\n",
      "0.6790616803954209\n",
      "715\n",
      "0.7964454151311307\n",
      "716\n",
      "0.7220465454644567\n",
      "717\n",
      "0.7730794098178032\n",
      "718\n",
      "0.8329577223144363\n",
      "719\n",
      "0.7076849131818099\n",
      "720\n",
      "0.6488178279985185\n",
      "721\n",
      "0.6965758419242525\n",
      "722\n",
      "0.7715002076856551\n",
      "723\n",
      "0.7119781587366921\n",
      "724\n",
      "0.6317526615784818\n",
      "725\n",
      "0.7268267938455942\n",
      "726\n",
      "0.6074360126649311\n",
      "727\n",
      "0.7028669626342737\n",
      "728\n",
      "0.7951226692369789\n",
      "729\n",
      "0.6509957685795623\n",
      "730\n",
      "0.8034751965221286\n",
      "731\n",
      "0.7986274954192945\n",
      "732\n",
      "0.7941241199821565\n",
      "733\n",
      "0.5704818468158427\n",
      "734\n",
      "0.6515370503070851\n",
      "735\n",
      "0.7304429414898576\n",
      "736\n",
      "0.6818200243938262\n",
      "737\n",
      "0.712766128213228\n",
      "738\n",
      "0.5674990850999975\n",
      "739\n",
      "0.7145323852737515\n",
      "740\n",
      "0.7021136984190872\n",
      "741\n",
      "0.6662963805891097\n",
      "742\n",
      "0.5916544080019523\n",
      "743\n",
      "0.6571254256696614\n",
      "744\n",
      "0.6259060095923021\n",
      "745\n",
      "0.7568125396748477\n",
      "746\n",
      "0.6806850282023484\n",
      "747\n",
      "0.697632553593744\n",
      "748\n",
      "0.5841827039703239\n",
      "749\n",
      "0.6664245826222394\n",
      "750\n",
      "0.7152562975432718\n",
      "751\n",
      "0.640767279026654\n",
      "752\n",
      "0.7021818751417025\n",
      "753\n",
      "0.6298434588116746\n",
      "754\n",
      "0.6272356912085794\n",
      "755\n",
      "0.8460949496970257\n",
      "756\n",
      "0.657605087709819\n",
      "757\n",
      "0.6908060695525762\n",
      "758\n",
      "0.5674913276871282\n",
      "759\n",
      "0.6640924822116271\n",
      "760\n",
      "0.7733998173490368\n",
      "761\n",
      "0.5793575757116102\n",
      "762\n",
      "0.6032995886032514\n",
      "763\n",
      "0.6298763550877399\n",
      "764\n",
      "0.6350804764471856\n",
      "765\n",
      "0.6664361805119178\n",
      "766\n",
      "0.662683472572722\n",
      "767\n",
      "0.668469454527596\n",
      "768\n",
      "0.7604427089704767\n",
      "769\n",
      "0.6057820753512491\n",
      "770\n",
      "0.7574561878904974\n",
      "771\n",
      "0.5568363450739165\n",
      "772\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7637578944792363\n",
      "773\n",
      "0.7056193936960078\n",
      "774\n",
      "0.5721533356612769\n",
      "775\n",
      "0.5788549067294341\n",
      "776\n",
      "0.6218258013065581\n",
      "777\n",
      "0.6270442181916698\n",
      "778\n",
      "0.7105147557993834\n",
      "779\n",
      "0.6589875006943028\n",
      "780\n",
      "0.563346506570164\n",
      "781\n",
      "0.6581825603374938\n",
      "782\n",
      "0.7792126734643765\n",
      "783\n",
      "0.7517388910474588\n",
      "784\n",
      "0.6030386882741425\n",
      "785\n",
      "0.6324810117452525\n",
      "786\n",
      "0.6734558985787161\n",
      "787\n",
      "0.69794729987398\n",
      "788\n",
      "0.5649429583333689\n",
      "789\n",
      "0.6487376564292903\n",
      "790\n",
      "0.6183941295664409\n",
      "791\n",
      "0.5810627148171635\n",
      "792\n",
      "0.6866751529627948\n",
      "793\n",
      "0.5453502092189342\n",
      "794\n",
      "0.5853727641124787\n",
      "795\n",
      "0.6762944921172792\n",
      "796\n",
      "0.6146650614197198\n",
      "797\n",
      "0.6114307406213478\n",
      "798\n",
      "0.6267021383988564\n",
      "799\n",
      "0.6436131336012744\n",
      "800\n",
      "0.6879246282373513\n",
      "801\n",
      "0.6218280585803495\n",
      "802\n",
      "0.663400240363309\n",
      "803\n",
      "0.6588913874708558\n",
      "804\n",
      "0.6942946987931674\n",
      "805\n",
      "0.5905432750915833\n",
      "806\n",
      "0.6103810324365585\n",
      "807\n",
      "0.6226374611658145\n",
      "808\n",
      "0.6462259467095172\n",
      "809\n",
      "0.592622777305129\n",
      "810\n",
      "0.6005512552963692\n",
      "811\n",
      "0.670331552444015\n",
      "812\n",
      "0.5338808981066797\n",
      "813\n",
      "0.8504065950764956\n",
      "814\n",
      "0.5056301389253332\n",
      "815\n",
      "0.6512027668689102\n",
      "816\n",
      "0.5581442745491434\n",
      "817\n",
      "0.4781721444688912\n",
      "818\n",
      "0.6556499240011193\n",
      "819\n",
      "0.5840469786889012\n",
      "820\n",
      "0.5363381456198535\n",
      "821\n",
      "0.6014735430782\n",
      "822\n",
      "0.6423078098286956\n",
      "823\n",
      "0.4798342383964277\n",
      "824\n",
      "0.6197101837155305\n",
      "825\n",
      "0.6030971734526283\n",
      "826\n",
      "0.6404909838315673\n",
      "827\n",
      "0.7198156586602888\n",
      "828\n",
      "0.5870086777787927\n",
      "829\n",
      "0.5793723548270036\n",
      "830\n",
      "0.5822281220593352\n",
      "831\n",
      "0.5120616486398517\n",
      "832\n",
      "0.6471851009471087\n",
      "833\n",
      "0.6626425346215568\n",
      "834\n",
      "0.5902752623094125\n",
      "835\n",
      "0.614424613527272\n",
      "836\n",
      "0.5933134414001817\n",
      "837\n",
      "0.738017803380674\n",
      "838\n",
      "0.5928854770367102\n",
      "839\n",
      "0.5376462040396082\n",
      "840\n",
      "0.6489584909993167\n",
      "841\n",
      "0.6891668680696077\n",
      "842\n",
      "0.513460348719056\n",
      "843\n",
      "0.4830740233832146\n",
      "844\n",
      "0.6282794945335708\n",
      "845\n",
      "0.5782276360631252\n",
      "846\n",
      "0.5600580537415178\n",
      "847\n",
      "0.6685852044681079\n",
      "848\n",
      "0.5373652596049316\n",
      "849\n",
      "0.608774727299236\n",
      "850\n",
      "0.6944160395654374\n",
      "851\n",
      "0.5908564993288095\n",
      "852\n",
      "0.6008612769106844\n",
      "853\n",
      "0.5547419949881676\n",
      "854\n",
      "0.5511264790440294\n",
      "855\n",
      "0.5549497540681193\n",
      "856\n",
      "0.6616186686277123\n",
      "857\n",
      "0.7164385102492444\n",
      "858\n",
      "0.45820196535317415\n",
      "859\n",
      "0.6027902912077926\n",
      "860\n",
      "0.5140795969821983\n",
      "861\n",
      "0.5465196453049324\n",
      "862\n",
      "0.6675116123137864\n",
      "863\n",
      "0.7048741036669618\n",
      "864\n",
      "0.6138466657077098\n",
      "865\n",
      "0.4922594872968288\n",
      "866\n",
      "0.5944468641159933\n",
      "867\n",
      "0.5162885972854099\n",
      "868\n",
      "0.5669129446354314\n",
      "869\n",
      "0.6848881288199442\n",
      "870\n",
      "0.5654146511011309\n",
      "871\n",
      "0.5069195705931605\n",
      "872\n",
      "0.6635446575703334\n",
      "873\n",
      "0.4959719795787958\n",
      "874\n",
      "0.5545371732901626\n",
      "875\n",
      "0.772870770029302\n",
      "876\n",
      "0.5494986817667927\n",
      "877\n",
      "0.4837329520064855\n",
      "878\n",
      "0.5123219925085892\n",
      "879\n",
      "0.5850576308519986\n",
      "880\n",
      "0.6690968184688663\n",
      "881\n",
      "0.4995723585818173\n",
      "882\n",
      "0.539523413367861\n",
      "883\n",
      "0.6541689855467099\n",
      "884\n",
      "0.6579174861324202\n",
      "885\n",
      "0.5444487094812769\n",
      "886\n",
      "0.6294864213038576\n",
      "887\n",
      "0.5724452355842078\n",
      "888\n",
      "0.5697927739973959\n",
      "889\n",
      "0.491570031802296\n",
      "890\n",
      "0.5050438154958962\n",
      "891\n",
      "0.4771695204526737\n",
      "892\n",
      "0.6052829570265961\n",
      "893\n",
      "0.596675371337271\n",
      "894\n",
      "0.6322869969555395\n",
      "895\n",
      "0.5031796224540662\n",
      "896\n",
      "0.5143931854351991\n",
      "897\n",
      "0.5827265099412198\n",
      "898\n",
      "0.5448266749094727\n",
      "899\n",
      "0.5247905158543488\n",
      "900\n",
      "0.59320947194589\n",
      "901\n",
      "0.45850138771218435\n",
      "902\n",
      "0.6627323878910915\n",
      "903\n",
      "0.5129670493983896\n",
      "904\n",
      "0.6824559883890828\n",
      "905\n",
      "0.5490836214787377\n",
      "906\n",
      "0.586928911756846\n",
      "907\n",
      "0.5247246188413218\n",
      "908\n",
      "0.5737101575706544\n",
      "909\n",
      "0.5293140752694853\n",
      "910\n",
      "0.553762874753873\n",
      "911\n",
      "0.6254815929197994\n",
      "912\n",
      "0.4785830995744627\n",
      "913\n",
      "0.6209809576792601\n",
      "914\n",
      "0.5046600447249641\n",
      "915\n",
      "0.5281719014882883\n",
      "916\n",
      "0.5779190114936532\n",
      "917\n",
      "0.6012253857688571\n",
      "918\n",
      "0.6899885587186708\n",
      "919\n",
      "0.627048903553938\n",
      "920\n",
      "0.48256191118899694\n",
      "921\n",
      "0.5767468732552063\n",
      "922\n",
      "0.587165880574331\n",
      "923\n",
      "0.6804829635113159\n",
      "924\n",
      "0.6244272007087401\n",
      "925\n",
      "0.5876711598728506\n",
      "926\n",
      "0.5207553368774935\n",
      "927\n",
      "0.5210075365788811\n",
      "928\n",
      "0.5698365730140423\n",
      "929\n",
      "0.5642443972623137\n",
      "930\n",
      "0.4544767798092655\n",
      "931\n",
      "0.5369789316159327\n",
      "932\n",
      "0.6643706554144105\n",
      "933\n",
      "0.5260111710830346\n",
      "934\n",
      "0.6799176887078429\n",
      "935\n",
      "0.4475261600425404\n",
      "936\n",
      "0.4769505573144323\n",
      "937\n",
      "0.47877237909825765\n",
      "938\n",
      "0.49838589659395827\n",
      "939\n",
      "0.5452949242013119\n",
      "940\n",
      "0.6089425912024911\n",
      "941\n",
      "0.5121643546886845\n",
      "942\n",
      "0.42003170978721555\n",
      "943\n",
      "0.6124921142937113\n",
      "944\n",
      "0.5571028833268011\n",
      "945\n",
      "0.5591053585421633\n",
      "946\n",
      "0.38316007044186434\n",
      "947\n",
      "0.6766968526337367\n",
      "948\n",
      "0.5473048119232645\n",
      "949\n",
      "0.5210919605073483\n",
      "950\n",
      "0.5160372058692071\n",
      "951\n",
      "0.5186683083740851\n",
      "952\n",
      "0.45812892656612036\n",
      "953\n",
      "0.5425318660502627\n",
      "954\n",
      "0.5206845037735919\n",
      "955\n",
      "0.5559689098975591\n",
      "956\n",
      "0.5927477468068632\n",
      "957\n",
      "0.6104525291545856\n",
      "958\n",
      "0.7985715099332793\n",
      "959\n",
      "0.5921278020454256\n",
      "960\n",
      "0.48447711526733545\n",
      "961\n",
      "0.48211346857110415\n",
      "962\n",
      "0.5916024444755541\n",
      "963\n",
      "0.5488877300130757\n",
      "964\n",
      "0.4914776001933959\n",
      "965\n",
      "0.5901492169929607\n",
      "966\n",
      "0.5617207772740123\n",
      "967\n",
      "0.559318118371908\n",
      "968\n",
      "0.4891285547592583\n",
      "969\n",
      "0.6036181855183503\n",
      "970\n",
      "0.6662791503602858\n",
      "971\n",
      "0.5804655051829429\n",
      "972\n",
      "0.4727829941049466\n",
      "973\n",
      "0.553326308795751\n",
      "974\n",
      "0.48388755099192243\n",
      "975\n",
      "0.45266685087988306\n",
      "976\n",
      "0.500683670922317\n",
      "977\n",
      "0.5587322115867999\n",
      "978\n",
      "0.5217007100605396\n",
      "979\n",
      "0.5689636955151123\n",
      "980\n",
      "0.5451217982063062\n",
      "981\n",
      "0.6201106851082145\n",
      "982\n",
      "0.5530160485384934\n",
      "983\n",
      "0.4845419299138181\n",
      "984\n",
      "0.4799213018123351\n",
      "985\n",
      "0.5874193522527439\n",
      "986\n",
      "0.5102245749762137\n",
      "987\n",
      "0.6652658070355062\n",
      "988\n",
      "0.62133629641872\n",
      "989\n",
      "0.6506476688279674\n",
      "990\n",
      "0.5573711866482874\n",
      "991\n",
      "0.5368900359870771\n",
      "992\n",
      "0.5029351827963991\n",
      "993\n",
      "0.46891882377297733\n",
      "994\n",
      "0.6425067721413549\n",
      "995\n",
      "0.5181012088422934\n",
      "996\n",
      "0.5456217823544905\n",
      "997\n",
      "0.5328568506744985\n",
      "998\n",
      "0.5960006154191844\n",
      "999\n",
      "0.46897665308500336\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from dataset.mnist import load_mnist\n",
    "from two_layer_net import TwoLayerNet\n",
    "\n",
    "# 读入数据\n",
    "(x_train, t_train), (x_test, t_test) = load_mnist(normalize=True, one_hot_label=True)\n",
    "\n",
    "train_loss_list = []\n",
    "\n",
    "# 超参数\n",
    "iters_num = 10000 # 循环次数\n",
    "train_size = x_train.shape[0] # 训练数据的大小\n",
    "batch_size = 100 # 批数量\n",
    "learning_rate = 0.1 # 学习率\n",
    "\n",
    "network = TwoLayerNet(input_size=784, hidden_size=50, output_size=10)\n",
    "\n",
    "for i in range(iters_num):\n",
    "    print(i)\n",
    "    # 获取 mini-batch\n",
    "    batch_mask = np.random.choice(train_size, batch_size)\n",
    "    x_batch = x_train[batch_mask]\n",
    "    t_batch = t_train[batch_mask]\n",
    "    \n",
    "    # 计算梯度\n",
    "    # grad = network.numerical_gradient(x_batch, t_batch)\n",
    "    grad = network.gradient(x_batch, t_batch) # 高速版！\n",
    "    \n",
    "    # 更新参数\n",
    "    for key in ('W1', 'b1', 'W2', 'b2'):\n",
    "        network.params[key] -= learning_rate * grad[key]\n",
    "        \n",
    "    # 记录学习过程\n",
    "    loss = network.loss(x_batch, t_batch)\n",
    "    print(loss)\n",
    "    train_loss_list.append(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里，mini-batch 的大小为100，需要每次从60000个训练数据中随机取出100个数据（图像数据和正确解标签数据）。然后，对这个包含100笔数据的 mini-batch 求梯度，使用随机梯度下降法(SGD)更新参数。这里，梯度法的更新次数（循环的次数）为10000。每更新一次，都对训练数据计算损失函数的值，并把该值添加到数组中。\n",
    "\n",
    "可以发现，随着学习的进行，损失函数的值在不断减小。这是学习正常进行的信号，表示神经网络的权重参数在逐渐你和数据。也就是说，神经网络的确在学习。通过反复地向它输入数据，神经网络正在逐渐向最优参数靠近。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 基于测试数据的评价\n",
    "神经网络的学习中，必须确认是否能够正确识别训练数据以外的其他数据，即确认是否会发生过拟合。过拟合是指，虽然训练数据中的数字图像能被正确辨认，但是不在训练数据中的数字图像却无法被识别的现象。\n",
    "\n",
    "神经网络学习的最初目标是掌握泛化能力，因此，要评价神经网络的泛化能力，就必须使用不包含在训练数据中的数据。下面的代码在进行学习的过程中，会定期地对训练数据和测试数据记录识别精度。这里，每经过一个 epoch，我们都会记录下训练数据和测试数据的识别精度。\n",
    "\n",
    "**epoch** 是一个单位。一个 epoch 表示学习中所有训练数据均被使用过一次时的更新次数。比如，对于10000笔训练数据，用大小为100笔数据的 mini-batch 进行学习时，重复随机梯度下降法100次，所有的训练数据就都被“看过”了。此时，100次就是一个 epoch。\n",
    "\n",
    "为了正确地评价，我们稍微修改一下前面的代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train acc, test acc | 0.10441666666666667, 0.1028\n",
      "train acc, test acc | 0.7982666666666667, 0.7978\n",
      "train acc, test acc | 0.8788666666666667, 0.881\n",
      "train acc, test acc | 0.8981833333333333, 0.9023\n",
      "train acc, test acc | 0.90685, 0.9098\n",
      "train acc, test acc | 0.9126833333333333, 0.9146\n",
      "train acc, test acc | 0.9173833333333333, 0.9188\n",
      "train acc, test acc | 0.9221333333333334, 0.9248\n",
      "train acc, test acc | 0.9255333333333333, 0.9258\n",
      "train acc, test acc | 0.9280666666666667, 0.9305\n",
      "train acc, test acc | 0.9309, 0.9329\n",
      "train acc, test acc | 0.9337333333333333, 0.9352\n",
      "train acc, test acc | 0.93645, 0.9384\n",
      "train acc, test acc | 0.9383833333333333, 0.9404\n",
      "train acc, test acc | 0.9402333333333334, 0.9406\n",
      "train acc, test acc | 0.94255, 0.9427\n",
      "train acc, test acc | 0.9449666666666666, 0.9443\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3de5xVdb3/8ddnX+c+DMxAwKDg/ZaAgqmpaZYC3jXNUjM7SZ7UrE4eL5WX6pxjWmkdSzNFzfxp3tIsU9O8nFJSMBTFC2giA8gMMMDc9+3z+2NvaBgG2KOzZw2z38/HYz9m9lprr/2eGVifvdb6XszdERGR4hUKOoCIiARLhUBEpMipEIiIFDkVAhGRIqdCICJS5FQIRESKXMEKgZnNMrNGM3t1M+vNzH5mZovM7BUz26dQWUREZPMKeUZwGzBtC+unAzvnHjOBGwqYRURENqNghcDdnwVWb2GT44Bfe9ZsYJiZjS5UHhER6V0kwPceCyzp9rwht2x5zw3NbCbZswbKy8v33W233QYkoIjIUDF37tyV7l7X27ogC4H1sqzX8S7c/SbgJoApU6b4nDlzCplLRGTIMbPFm1sXZKuhBmBct+f1wLKAsoiIFK0gC8HvgS/kWg/tD6x1900uC4mISGEV7NKQmd0FHArUmlkDcDkQBXD3G4FHgBnAIqAdOKtQWUREZPMKVgjc/XNbWe/AuYV6fxERyY96FouIFDkVAhGRIqdCICJS5FQIRESKnAqBiEiRC7JnsYhIUcmkMySSXXQlEiQTCRKpJF2RahKpDJl1K0h3riGZTJBOJkglkyTS0FS5O4l0hti699h+pz2YMn54v+dSIRCRYLhDJg1mEApDKgFtTZBOQCaV/ZpOQM14KK2BtpWw9KV/LU8ns193+hRUjSbdtJDMW4+RTmfIOKQzGTLurNvxOLpK6wg1LqDkvafJZDy7LuNkPMN7259ER7SG8pUvM3zF82QyGTy3b08leG7sl2izMsY3Pc2uq5/EMkkskySUThLyJNfWXkFrJs6R6+7jsPbHCHuSiKeyD5IcZrfQlYZL/VecFn6CEqAk9yto9zh7dN0KwHXR6zk+/NxGv6Imr+YLXdmBmb8afoi3kt9QIRCRAnDPHozd/3UgTnX962t5LVTXZw/Ubz9JJtFBqqudVFcb6UQHbSP3paV2EqmWJmpf/BGe7IBkB6Q6sFQnb213Ku/UfZLS5rc4ct65RNKdhDNdRDNdGM7dYy/l71VHMq71Zb655GubxPufykt5NnIgk7pe5H/ar9xk/RfTl/JMai+m22x+EftZttdqN2f+OcQ834mTw09zTfSmTV5/zgsjWej1nBX+E5dH79iwPOUhkkQ4d9FUGkN1fDHyBgeEXiZFhJRFSVuUtEVY29ZFOhqlLVzN+7Ht8XCMTCiKh6IQinLMuI8QicaIt0zj+fYJWDhKKBIlFI5gkRKu3WkisXCYkc3n8XrnSYQjMcLRKJFIlFC8nKe3P4R4NERJ+x6U1m73of/cvbFsv65thwadkyFn/YEYYNXb0L4KulpyB+Ou7KfhHQ7Nrn/p19DWhCe7SCW7SCc76Rq2E2t2/zxdqQy1T34L61i50YH8/doDmLfTV+lKpjn+r8cSTbURyiQJe4JwJsnsmmO4e+TXSSa6uPGfR24S7zfh47mW04mn1vGcfWmT9T9Ofob/TZ/IKFbzx/ildBKj02PZr8S4JTWdP2U+xihW863IPXQQJxmKkQrFyYRi/C00lcWxHRhp6zg482L2IBqOQiiGh6MsKdmV9ngdVbQxJr0UC8ewSBSLxLFwjK6S4YQipcQtTal1EQ2FiISNSDhEJBwiFCsjHI0RtxQxTxGJhIiGQ0TC4ew2sRKikQgRSxM1JxIyYrEYsViUWDhELBwiFOptjMxti5nNdfcpva5TIZCi5Z49YGaSEKvIHoy7WiDRlr1k4ensVxyG75B9zdoGaF9NJp0ikUzQlUjSlYZ1tZPoSGSILHuR8Jp38UQLdLZgiVa6LMYrO5xNVzLDlDd+yMg1LxNNtRFLZx+N8fFcO/5GulIZ/nPx2WyfWLRRzHmhvTg39n26UmnuS57PeMsOyZX0MAkiPJnZh68lzwfgt7HvUUkHCSJ0ESXhEf6W2Ysb08cC8F+RWzCclEXIhGJkQjHejOzKi/H9iUdCHJ96BMIxPBLHwnEsEmdN2XasLd+B0giM61pEKFpKKF5KOFZKOFZOOF5GLB4nHgkTi4SIb3iEiUc3/T4WDmG27R9YtzUqBDK0pVNgIQiFsgfqxjegoxk6mkm1N5NqXUXT1P9kXSZK+bxZ1L7xGyJda4gm1xHOJAD44dT/oz0V4ojFP+Ljq3+30e6TRDiy4n46kmku7fopx/DMRuubvYLJXdlLDr+IXseM8Av/eq2HWeRjmJ74IQCXRu5kZ2ugjVI6rIyOUBkrQqN4KH408UiIKf4aZeEUqUgZREoIRUtIRyvpKB1NSTREpXUSjUWJREuJxyLEIyFKouF/HWzXP88ddEuiYWLhf21TEs0erMND4BOu9I0KgQw+6RQkWrOfvhNtUDUa4pWwtoHMu8+R6Ggh0b6OZMc60p2tLN7xDFZF6ih772l2WjSLSGItseQ6SpLrKMm0ccnY23g78xGOWHMPX+6YtdFbtXoJh3f9iBUM59jQ35gWfpG1Xs5aKljnZSSI8P9sBrFYjP3Db7JrqIFQOEIkEiESjhCKRHll+JGURkPsmHyL2sxKIpEo0Wg0e0khVkLL6AMoiYSoTrxPzNJESqsIl1YRi5cSj0Y2fFKO5T4RR8JquS0DS4VA+k86Ba0rspdQulqga13260c+CiN2JN38Hsm/3UCyfS2pjnV4Z3b9Szuew6KKKQxvnM0JC75O1BMb7faSsst5KjWRj3U9x0/tRxut6/AYpyUu5SXfhcNC/+DcyEOs9XJarIKOcCUdkSr+Un4UydI6tousZmyoGUqHYWU1RMtqKCstpaIkSkU8QmVJhIp4hIqSCJXxCOXxCKXR8JC4BiyyJVsqBGo1VKzSqexNybam7KNyNIzcDTrWwOPfIdXaRLqlCe9chyVaeGunL/PymFNg1SJOf/HETXb3o8hMbk9+ivrE29wXu4V2Smn1UlpyX295fzHPZyoYb120xqeRCpeRiZaTiZbjsXK8ai8OKa9leGQat9rHiJZVEi+tIlZeSWVpnIvj2QN5RfwwyuMXUlESIR4Jb3j/TW9hiki+dEYwFL03O/upva0p2/a6bSWMmQSTT6e1vYPSn+5GuGvNRi/5y/BTuKnkS6xbt5bbWs5hpVexyitZRxmtXsafMvvxdGYSZXRyYvR5MrFKiFViJZWES6tIVYwhVjGc6tLoRo9hZdmvVaVRKkuyn751o1Bk4OnS0FCVTsGK+bD4OQjHWPvRL7Jg2Tr2+e0U4onVGzZrsUoejRzG5V2n0Z5I853IHbR4GauoYqVXsy5UTWfFdoSqxzCyKs7IyhJGVsWpq4hTUxajumzjg3tJNLyFUCIyGOnS0FDz0q/xBb8ns/h5wslWAOaGJ3LSA2MA2NfOp50S2iI1RCtrGVFVwciqOKfmDvAjqn7E7pUljKqKU1dZQlVJRJ/SRYqYCsFglmiHpXNg8XMklr7M3/a5jn8sWcM+8/5IfdvrzE5/jL9ndufN+F6Mrd+R/9iuhr3HDaO+5hOMrIxTEdcBXkS2ToVgEEq/+Rhdf7mGeOM/CHuKDMZbme35j/l/YY1Vsceof2fSLiOYPK6Gb25fw/gRZTrgi8gHpkIQtPbVdDz9ExKLnubhMV/nj6vrqW6Yy0yaeSEznddje5Gp35/dd6jn5+Nq2Lu+mvK4/mwi0n90RAlSayOrb5xBZcs7vOI788cVi2kb/RF22fdYlmx/JjPG1fCV4aX6tC8iBaVCEBBfs4TmG6dT0tHI/47+Hw468mRmja2mNKYWOSIysFQIApDOOL+/73YO71jJr3f6CRecdprGfhGRwKgQDLCurk6+ed8C/rhoMhceeD9fPeZAXfoRkUBp5KsB1LF4LmuvnsiSV//Kt2fszrnHflxFQEQCpzOCAbLurb8Svutkkpkyzpm2LzMO2SHoSCIigArBgFg1/3HK7j+d5T6cJUffxYypk4OOJCKygQpBgTW8+lfq7v88i300Lafcyyf23C3oSCIiG9E9ggKa37CWEx9o4U47mvQX/sC+KgIiMgjpjKBA3nzyds57toRoaR2HffnnTKgtDzqSiEivdEZQAAsevIZd/+9rfD3+MPf/+4EqAiIyqOmMoJ+9ctdl7P3mT5kdP5DDzr2BYVUlQUcSEdkiFYL+4s5Lt36Dfd67lb+VHc7kr/0/ykpUBERk8NOloX7g7vz4D3Opevcx/lp9NFO/cY+KgIhsM3RG8CGlkkm+8+Ar3D13BZ1TbuHiE/YnHFZ9FZFtR0ELgZlNA34KhIGb3f2qHuurgd8A2+Wy/Mjdby1kpv7U2dnB/P89lYPWdTLykz/jG5/eRUNGiMg2p2AfXc0sDPwcmA7sAXzOzPbosdm5wAJ3nwgcCvzYzGKFytSfWlpbeO3a45ja9jSjdjuQbx6xq4qAiGyTCnkNYz9gkbu/4+4J4G7guB7bOFBp2SNoBbAaSBUwU79oaVnHO9fNYHLnC8ybeDlTT7s86EgiIh9YIQvBWGBJt+cNuWXdXQ/sDiwD5gMXuHum547MbKaZzTGzOU1NTYXKm7c3nr2XialXeHWf7zHphG8GHUdE5EMpZCHo7TqJ93h+JDAPGANMAq43s6pNXuR+k7tPcfcpdXV1/Z+0jxYnh3FX6jB2PvzMoKOIiHxohSwEDcC4bs/ryX7y7+4s4AHPWgT8Exj0A/LMSe/Mj0vOpbSiOugoIiIfWiELwYvAzmY2IXcD+FTg9z22eQ84HMDMRgG7Au8UMFO/aF61gvph6icgIkNDwZqPunvKzM4DHiPbfHSWu79mZufk1t8IfB+4zczmk72UdJG7ryxUpv5y6fILWFm+M3Bw0FFERD60gvYjcPdHgEd6LLux2/fLgCMKmaG/pdMZRmUaeb/y0KCjiIj0C3WB7aOVK5ZQYkls2PZBRxER6RcqBH20qmERACV144MNIiLST1QI+qh1xdsADBuzU8BJRET6hwpBHy30cfwk+RlGjts56CgiIv1ChaCPXk6M4a6yz1FSvkm/NxGRbZIKQR9lGt9kj+pE0DFERPqN5iPoowuaLmNlxa7AjKCjiIj0C50R9EEqlWJUpolExbitbywiso1QIeiDpveXELMUVqM+BCIydKgQ9MGqhoUAlI2cEHASEZH+o0LQB20rsuPhqQ+BiAwlKgR98Fp4Ny5MzqRunAqBiAwdajXUBws6avi/8iOJl1YEHUVEpN+oEPRBxfK/s3+lOpKJyNCiQtAH/9b8E5oqdiM7x46IyNCgewR5SiWTjMo0kqxUHwIRGVpUCPLUuGwxMUsTHq4+BCIytKgQ5GnVsuw8BKUjdwg4iYhI/1IhyFP7hnkIdgw4iYhI/1IhyNPc2FROT1xC3bhdg44iItKvVAjytKglxjuVU4nFS4KOIiLSr1QI8jR2+eN8uvztoGOIiPQ7FYI8nbrmZo5LPRp0DBGRfqdCkIdEIslIX0mySn0IRGToUSHIQ+Oyd4hamrDmIRCRIUiFIA/NS7P3BtSHQESGIhWCPLQ3ZuchGD5Ww0+LyNCjQpCH50sP4VOJH1M3bpego4iI9DsVgjy8uyZNR9UORKKxoKOIiPQ7DUOdhz2X3st2JcOATwYdRUSk36kQ5OGolnt5v2rvoGOIiBSELg1tRVeiK9uHoHq7oKOIiBSECsFWNDa8S8Qy6kMgIkOWCsFWNC97C4CykRMCTiIiUhgFLQRmNs3M3jSzRWZ28Wa2OdTM5pnZa2b2TCHzfBCtTUsAGF6vpqMiMjQVrBCYWRj4OTAd2AP4nJnt0WObYcAvgGPdfU/g5ELl+aD+WvpJPpq4ldr6nYOOIiJSEIU8I9gPWOTu77h7ArgbOK7HNp8HHnD39wDcvbGAeT6QhuYOhg0bRiSiBlYiMjQVshCMBZZ0e96QW9bdLkCNmT1tZnPN7Au97cjMZprZHDOb09TUVKC4vfv4kpv4YvTJAX1PEZGBVMhCYL0s8x7PI8C+wFHAkcB3zWyTi/HufpO7T3H3KXV1df2fdAsOaX+cSbZwQN9TRGQg5VUIzOx+MzvKzPpSOBqA7gP41wPLetnmUXdvc/eVwLPAxD68R0F1dnYy0leRqtQ8BCIydOV7YL+B7PX8hWZ2lZntlsdrXgR2NrMJZhYDTgV+32Obh4CDzSxiZmXAx4DX88xUcCuWvkPYnMgI9SEQkaErr0Lg7k+4+2nAPsC7wJ/N7DkzO8vMopt5TQo4D3iM7MH9Hnd/zczOMbNzctu8DjwKvAK8ANzs7q9+2B+qv6xZtgiA8lGah0BEhq68m8KY2QjgdOAM4B/AncBBwJnAob29xt0fAR7psezGHs+vAa7pS+iB0rx6Fc1ewfAxmodARIaufO8RPAD8H1AGHOPux7r7b939fKCikAGDNDt2APulfsWIcbsGHUVEpGDyPSO43t3/0tsKd5/Sj3kGlYbmdsYMKyUc6q0BlIjI0JDvzeLdc72AATCzGjP7aoEyDRpHvXcN54cfCDqGiEhB5VsIznb3NeufuHszcHZhIg0ekzv/zoTQoOvsLCLSr/ItBCEz23B9JDeO0JCet7Gjo4ORvpp0lfoQiMjQlm8heAy4x8wON7NPAneRbfY5ZK1oWETInMiI8UFHEREpqHxvFl8EfAX4d7JDRzwO3FyoUIPBmmVvA+pDICJDX16FwN0zZHsX31DYOIPHynUdLMhsT52GnxaRIS7ffgQ7m9l9ZrbAzN5Z/yh0uCC9GJ7E8ZkfMkKdyURkiMv3HsGtZM8GUsBhwK+BOwoVajBoaO6gflgpIfUhEJEhLt9CUOruTwLm7ovd/Qrgk4WLFbzTF1/KxcwKOoaISMHle7O4MzcE9UIzOw9YCowsXKzg7ZB4kyXlNUHHEBEpuHzPCL5Odpyhr5GdSOZ0soPNDUltbW3UeTMZ9SEQkSKw1TOCXOexU9z9QqAVOKvgqQLWuPRtJpgTVR8CESkCWz0jcPc0sG/3nsVD3Zql6+chmBBwEhGRwsv3HsE/gIfM7F6gbf1Cdx+SI7It7wixOj2ZiePymYhNRGTblm8hGA6sYuOWQg4MyULwD9+FX/tFvDF6fNBRREQKLt+exUP+vkB3S1e3UV9TShFdDRORIpZXITCzW8meAWzE3b/U74kGgXMXf42W2Eg2MwOniMiQku+loT90+74EOAFY1v9xBoe61HI6qnSjWESKQ76Xhu7v/tzM7gKeKEiigLW0tjCSZt6t3i7oKCIiAyLfDmU97QwMySNl45Js09HoiO0DTiIiMjDyvUfQwsb3CN4nO0fBkLN2eXYegopRGnVURIpDvpeGKgsdZLBYkihnQepwpm+nPgQiUhzynY/gBDOr7vZ8mJkdX7hYwXkltT3/ZTMZPrI+6CgiIgMi33sEl7v72vVP3H0NcHlhIgVrddNythsWUx8CESka+TYf7a1g5PvabcrMhkvIRMsZ4tMtiIhskO8ZwRwz+4mZ7WhmO5jZtcDcQgYLSl36fTrLxwQdQ0RkwORbCM4HEsBvgXuADuDcQoUKytp166hlLRn1IRCRIpJvq6E24OICZwlcY8NCqoHoCPUqFpHikW+roT+b2bBuz2vM7LHCxQrGumXZzmQVmodARIpIvpeGanMthQBw92aG4JzF72RGcXXys9SO/2jQUUREBky+hSBjZhsunJvZeHoZjXRb91pnHbeHT6R6xKigo4iIDJh8m4B+G/irmT2Te34IMLMwkYKTXPEWH60OqQ+BiBSVfG8WP2pmU8ge/OcBD5FtOTSkfOH9/yIZrQKOCzqKiMiAyfdm8ZeBJ4H/yD3uAK7I43XTzOxNM1tkZpttdWRmU80sbWafyS92/3N3alMr6KzQ0BIiUlzyvUdwATAVWOzuhwGTgaYtvcDMwsDPgenAHsDnzGyPzWz3QyDQVkhr165hhK3Dq8cFGUNEZMDlWwg63b0TwMzi7v4GsOtWXrMfsMjd33H3BHA3vV9zOR+4H2jMM0tBNDVkm47GRowPMoaIyIDLtxA05PoRPAj82cweYutTVY4FlnTfR27ZBmY2luy0lzduaUdmNtPM5pjZnKamLZ6IfGAb5iH4yI4F2b+IyGCVVyFw9xPcfY27XwF8F7gF2Now1L01venZ5PQ64CJ3T2/l/W9y9ynuPqWuri6fyH32JuP5euKr1E6YWJD9i4gMVn0eQdTdn9n6VkD2DKD7Bfd6Nj2LmALcnWuuWQvMMLOUuz/Y11wf1lvtFTwZPZRrhw0f6LcWEQlUIYeSfhHY2cwmAEuBU4HPd9/A3TeM5WBmtwF/CKIIAJQsf5GDqtSHQESKT8EKgbunzOw8sq2BwsAsd3/NzM7Jrd/ifYGBdmLT9SRjNcCZQUcRERlQBZ1cxt0fAR7psazXAuDuXyxkli1xd0amVvB2zV5BRRARCUy+rYaGtOY1zdRYCz5s+6CjiIgMOBUCoGnJQgBiteODDSIiEgAVAqDl/Wwfgir1IRCRIqRCAMwP7cGpie8wXH0IRKQIqRAA77RGWBDbm+rqYVvfWERkiFEhAOqW/pnjKl4POoaISCAK2nx0WzFt1R10lYwAvh50FBGRAVf0ZwTuzsj0CrrKNQ+BiBSnoi8Eq1evYpi14sM0D4GIFKeiLwTr5yGI107YypYiIkNT0ReCdcuzhaDqIzsEnEREJBhFXwj+EZvCIV3XUrvTvkFHEREJRNEXgvfWJmkpraeivCLoKCIigSj6QrDjkvs5o+z5oGOIiASm6PsRHLzmQTpLRgUdQ0QkMEV9RpDtQ9BIV8XYoKOIiASmqAvBypWNVFsbaB4CESliRV0I1s9DEK9THwIRKV5FXQjWNr4HQNVozUMgIsWrqAvBS/H92LXzNurUh0BEilhRF4KG5g4qyisoKykJOoqISGCKuvnopMWz2DUO8Omgo4iIBKaoC8GUlr/QXjo66BgiIoEq2ktDmXQmOw9BheYhEJHiVrSFYOWqRiqtQ30IRKToFW8hWPIWAPG68cEGEREJWNEWgtWrVtLk1VSP3inoKCIigSraQjAv8lGmdt3AyF32CzqKiEigirYQLFndQW1FnJJoOOgoIiKBKtrmo59491oOiAJ8KugoIiKBKtpCsGvbXNrK1XRURKQoLw2l0xlGZVaQ0DwEIiLFWQgaG5dTYZ2Y+hCIiBRnIVi1dBEA8bodAk4iIhK8ghYCM5tmZm+a2SIzu7iX9aeZ2Su5x3NmNrGQedZrWtPKK5kJVI/dZSDeTkRkUCtYITCzMPBzYDqwB/A5M9ujx2b/BD7h7nsD3wduKlSe7ubbLhyb+C/qdpw8EG8nIjKoFfKMYD9gkbu/4+4J4G7guO4buPtz7t6cezobGJBmPEtWtzOyUn0IRESgsM1HxwJLuj1vAD62he3/DfhTbyvMbCYwE2C77bb70MGOeedKjg476kMgIlLYMwLrZZn3uqHZYWQLwUW9rXf3m9x9irtPqaur+9DBxnW+RU0k8aH3IyIyFBSyEDQA47o9rweW9dzIzPYGbgaOc/dVBcwDQCqVZlSmkUSlOpOJiEBhC8GLwM5mNsHMYsCpwO+7b2Bm2wEPAGe4+1sFzLJB44qllFkXVjN+IN5ORGTQK9g9AndPmdl5wGNAGJjl7q+Z2Tm59TcClwEjgF+YGUDK3acUKhNk+xCMAUrrJhTybUREthkFHWvI3R8BHumx7MZu338Z+HIhM/T0fmuGZekp7FnfsyWriEhxKrpB515Nb8fPUt/kjQl7Bh1FRDYjmUzS0NBAZ2dn0FG2OSUlJdTX1xONRvN+TdEVgqWr2xhVWUI8oj4EIoNVQ0MDlZWVjB8/ntxlY8mDu7Nq1SoaGhqYMCH/y99FN9bQ59+5kBv5QdAxRGQLOjs7GTFihIpAH5kZI0aM6POZVNEVgprEcixeEXQMEdkKFYEP5oP83oqqECRTaT6SaSRZoT4EIiLrFVUhaHx/KaWWIFSjeQhEZPPWrFnDL37xiw/02hkzZrBmzZp+TlRYRVUIVjVk+6yVjlQfAhHZvC0VgnQ6vcXXPvLIIwwbNqwQsQqmqFoNNXTEmZs6kiO2G5BpD0SkH1z58GssWLauX/e5x5gqLj9m803IL774Yt5++20mTZrEpz/9aY466iiuvPJKRo8ezbx581iwYAHHH388S5YsobOzkwsuuICZM2cCMH78eObMmUNrayvTp0/noIMO4rnnnmPs2LE89NBDlJaWbvReDz/8MD/4wQ9IJBKMGDGCO++8k1GjRtHa2sr555/PnDlzMDMuv/xyTjrpJB599FEuvfRS0uk0tbW1PPnkkx/691FUheCN5EiuT5/J6dtrQhoR2byrrrqKV199lXnz5gHw9NNP88ILL/Dqq69uaJY5a9Yshg8fTkdHB1OnTuWkk05ixIgRG+1n4cKF3HXXXfzqV7/ilFNO4f777+f000/faJuDDjqI2bNnY2bcfPPNXH311fz4xz/m+9//PtXV1cyfPx+A5uZmmpqaOPvss3n22WeZMGECq1ev7peft6gKweqm9xlXFSEaLqorYiLbtC19ch9I++2330Zt83/2s5/xu9/9DoAlS5awcOHCTQrBhAkTmDRpEgD77rsv77777ib7bWho4LOf/SzLly8nkUhseI8nnniCu+++e8N2NTU1PPzwwxxyyCEbthk+fHi//GxFdUT8zOIruSXz7aBjiMg2qLy8fMP3Tz/9NE888QTPP/88L7/8MpMnT+617X48Ht/wfTgcJpVKbbLN+eefz3nnncf8+fP55S9/uWE/7r5JU9DelvWHoioENcn3aS0ZE3QMERnkKisraWlp2ez6tWvXUlNTQ1lZGW+88QazZ8/+wO+1du1axo4dC8Dtt9++YfkRRxzB9ddfv+F5c3MzBxxwAM888wz//Oc/Afrt0lDRFIJEMjsPQapy3NY3FpGiNmLECD7+8Y+z1157ceGFF26yftq0aaRSKfbee2+++93vsv/++3/g97riiis4+eSTOfjgg6mtrd2w/Dvf+Q7Nzc3stddeTDVInQIAAArxSURBVJw4kaeeeoq6ujpuuukmTjzxRCZOnMhnP/vZD/y+3Zl7r5OGDVpTpkzxOXPm9Pl1S977J+NmTeKlvb7NPp/5zwIkE5H+8vrrr7P77rsHHWOb1dvvz8zmbm6Y/6I5I1i9dBGgeQhERHoqmkLQFhvJL2JfpHqHfYKOIiIyqBRN89ED953Igfv+NOgYIiKDTtGcEYiISO9UCEREipwKgYhIkVMhEBHp4cMMQw1w3XXX0d7e3o+JCkuFQESkh2IrBEXTakhEtmG3HrXpsj2Ph/3OhkQ73HnypusnfR4mnwZtq+CeL2y87qw/bvHteg5Dfc0113DNNddwzz330NXVxQknnMCVV15JW1sbp5xyCg0NDaTTab773e+yYsUKli1bxmGHHUZtbS1PPfXURvv+3ve+x8MPP0xHRwcHHnggv/zlLzEzFi1axDnnnENTUxPhcJh7772XHXfckauvvpo77riDUCjE9OnTueqqq/r629sqFQIRkR56DkP9+OOPs3DhQl544QXcnWOPPZZnn32WpqYmxowZwx//mC0sa9eupbq6mp/85Cc89dRTGw0Zsd55553HZZddBsAZZ5zBH/7wB4455hhOO+00Lr74Yk444QQ6OzvJZDL86U9/4sEHH+Tvf/87ZWVl/Ta2UE8qBCIy+G3pE3ysbMvry0ds9Qxgax5//HEef/xxJk+eDEBraysLFy7k4IMP5lvf+hYXXXQRRx99NAcffPBW9/XUU09x9dVX097ezurVq9lzzz059NBDWbp0KSeccAIAJSUlQHYo6rPOOouysjKg/4ad7kmFQERkK9ydSy65hK985SubrJs7dy6PPPIIl1xyCUccccSGT/u96ezs5Ktf/Spz5sxh3LhxXHHFFXR2drK5Md8KNex0T7pZLCLSQ89hqI888khmzZpFa2srAEuXLqWxsZFly5ZRVlbG6aefzre+9S1eeumlXl+/3vq5Bmpra2ltbeW+++4DoKqqivr6eh588EEAurq6aG9v54gjjmDWrFkbbjzr0pCIyADpPgz19OnTueaaa3j99dc54IADAKioqOA3v/kNixYt4sILLyQUChGNRrnhhhsAmDlzJtOnT2f06NEb3SweNmwYZ599Nh/96EcZP348U6dO3bDujjvu4Ctf+QqXXXYZ0WiUe++9l2nTpjFv3jymTJlCLBZjxowZ/Pd//3e//7xFMwy1iGw7NAz1h6NhqEVEpE9UCEREipwKgYgMStvaZevB4oP83lQIRGTQKSkpYdWqVSoGfeTurFq1akM/hHyp1ZCIDDr19fU0NDTQ1NQUdJRtTklJCfX19X16jQqBiAw60WiUCRM0v/hAKeilITObZmZvmtkiM7u4l/VmZj/LrX/FzDShsIjIACtYITCzMPBzYDqwB/A5M9ujx2bTgZ1zj5nADYXKIyIivSvkGcF+wCJ3f8fdE8DdwHE9tjkO+LVnzQaGmdnoAmYSEZEeCnmPYCywpNvzBuBjeWwzFljefSMzm0n2jAGg1cze/ICZaoGVH/C1hTRYc8HgzaZcfaNcfTMUc22/uRWFLAS9DZnXsy1YPtvg7jcBN33oQGZzNtfFOkiDNRcM3mzK1TfK1TfFlquQl4YagHHdntcDyz7ANiIiUkCFLAQvAjub2QQziwGnAr/vsc3vgS/kWg/tD6x19+U9dyQiIoVTsEtD7p4ys/OAx4AwMMvdXzOzc3LrbwQeAWYAi4B24KxC5cn50JeXCmSw5oLBm025+ka5+qaocm1zw1CLiEj/0lhDIiJFToVARKTIFU0h2NpwF0Ews3Fm9pSZvW5mr5nZBUFn6s7Mwmb2DzP7Q9BZ1jOzYWZ2n5m9kfu9HRB0JgAz+0bub/iqmd1lZn0b/rH/cswys0Yze7XbsuFm9mczW5j7WjNIcl2T+zu+Yma/M7NhgyFXt3XfMjM3s9qBzrWlbGZ2fu5Y9pqZXd0f71UUhSDP4S6CkAL+w913B/YHzh0kuda7AHg96BA9/BR41N13AyYyCPKZ2Vjga8AUd9+LbOOIUwOKcxswrceyi4En3X1n4Mnc84F2G5vm+jOwl7vvDbwFXDLQoeg9F2Y2Dvg08N5AB+rmNnpkM7PDyI7IsLe77wn8qD/eqCgKAfkNdzHg3H25u7+U+76F7EFtbLCpssysHjgKuDnoLOuZWRVwCHALgLsn3H1NsKk2iAClZhYBygioP4y7Pwus7rH4OOD23Pe3A8cPaCh6z+Xuj7t7Kvd0Ntl+RIHnyrkW+E966eA6UDaT7d+Bq9y9K7dNY3+8V7EUgs0NZTFomNl4YDLw92CTbHAd2f8ImaCDdLMD0ATcmrtkdbOZlQcdyt2Xkv1k9h7Z4VHWuvvjwabayKj1/XNyX0cGnKc3XwL+FHQIADM7Fljq7i8HnaUXuwAHm9nfzewZM5vaHzstlkKQ11AWQTGzCuB+4Ovuvm4Q5DkaaHT3uUFn6SEC7APc4O6TgTaCucyxkdw19+OACcAYoNzMTg821bbDzL5N9jLpnYMgSxnwbeCyoLNsRgSoIXsp+ULgHjPr7fjWJ8VSCAbtUBZmFiVbBO509weCzpPzceBYM3uX7GW0T5rZb4KNBGT/jg3uvv6s6T6yhSFonwL+6e5N7p4EHgAODDhTdyvWj+qb+9ovlxP6g5mdCRwNnOaDo1PTjmQL+su5f//1wEtm9pFAU/1LA/BAbsTmF8iesX/om9nFUgjyGe5iwOUq+S3A6+7+k6DzrOful7h7vbuPJ/u7+ou7B/4J193fB5aY2a65RYcDCwKMtN57wP5mVpb7mx7OILiJ3c3vgTNz358JPBRglg3MbBpwEXCsu7cHnQfA3ee7+0h3H5/7998A7JP7tzcYPAh8EsDMdgFi9MMoqUVRCHI3pNYPd/E6cI+7vxZsKiD7yfsMsp+45+UeM4IONcidD9xpZq8Ak4D/DjgPuTOU+4CXgPlk/18FMkSBmd0FPA/samYNZvZvwFXAp81sIdmWMFcNklzXA5XAn3P/9m8cJLkGhc1kmwXskGtSejdwZn+cSWmICRGRIlcUZwQiIrJ5KgQiIkVOhUBEpMipEIiIFDkVAhGRIqdCIFJgZnboYBrBVaQnFQIRkSKnQiCSY2anm9kLuc5Nv8zNx9BqZj82s5fM7Ekzq8ttO8nMZncbS78mt3wnM3vCzF7OvWbH3O4rus2jcOf68WHM7CozW5DbT78MKSzSVyoEIoCZ7Q58Fvi4u08C0sBpQDnwkrvvAzwDXJ57ya+Bi3Jj6c/vtvxO4OfuPpHseEPLc8snA18nOx/GDsDHzWw4cAKwZ24/PyjsTynSOxUCkazDgX2BF81sXu75DmQH9fptbpvfAAeZWTUwzN2fyS2/HTjEzCqBse7+OwB37+w2hs4L7t7g7hlgHjAeWAd0Ajeb2YnAoBhvR4qPCoFIlgG3u/uk3GNXd7+il+22NCbLloYD7ur2fRqI5MbA2o/s6LPHA4/2MbNIv1AhEMl6EviMmY2EDfP8bk/2/8hnctt8Hviru68Fms3s4NzyM4BncnNJNJjZ8bl9xHPj2/cqNw9Ftbs/Qvay0aRC/GAiWxMJOoDIYODuC8zsO8DjZhYCksC5ZCe/2dPM5gJryd5HgOxwzjfmDvTvAGfllp8B/NLMvpfbx8lbeNtK4CHLTnRvwDf6+ccSyYtGHxXZAjNrdfeKoHOIFJIuDYmIFDmdEYiIFDmdEYiIFDkVAhGRIqdCICJS5FQIRESKnAqBiEiR+//Zty0Qk5eKZQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import sys, os\n",
    "sys.path.append(os.pardir)  # 为了导入父目录的文件而进行的设定\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from dataset.mnist import load_mnist\n",
    "from two_layer_net import TwoLayerNet\n",
    "\n",
    "# 读入数据\n",
    "(x_train, t_train), (x_test, t_test) = load_mnist(normalize=True, one_hot_label=True)\n",
    "\n",
    "train_loss_list = []\n",
    "train_acc_list = []\n",
    "test_acc_list = []\n",
    "# 平均每个epoch的重复次数\n",
    "iter_per_epoch = max(train_size / batch_size, 1)\n",
    "\n",
    "# 超参数\n",
    "iters_num = 10000  # 适当设定循环的次数\n",
    "train_size = x_train.shape[0]\n",
    "batch_size = 100\n",
    "learning_rate = 0.1\n",
    "\n",
    "network = TwoLayerNet(input_size=784, hidden_size=50, output_size=10)\n",
    "\n",
    "for i in range(iters_num):\n",
    "    # 获取mini-batch\n",
    "    batch_mask = np.random.choice(train_size, batch_size)\n",
    "    x_batch = x_train[batch_mask]\n",
    "    t_batch = t_train[batch_mask]\n",
    "    \n",
    "    # 计算梯度\n",
    "    #grad = network.numerical_gradient(x_batch, t_batch)\n",
    "    grad = network.gradient(x_batch, t_batch) # 高速版！\n",
    "    \n",
    "    # 更新参数\n",
    "    for key in ('W1', 'b1', 'W2', 'b2'):\n",
    "        network.params[key] -= learning_rate * grad[key]\n",
    "    \n",
    "    loss = network.loss(x_batch, t_batch)\n",
    "    train_loss_list.append(loss)\n",
    "    # 计算每个epoch的识别精度\n",
    "    if i % iter_per_epoch == 0:\n",
    "        train_acc = network.accuracy(x_train, t_train)\n",
    "        test_acc = network.accuracy(x_test, t_test)\n",
    "        train_acc_list.append(train_acc)\n",
    "        test_acc_list.append(test_acc)\n",
    "        print(\"train acc, test acc | \" + str(train_acc) + \", \" + str(test_acc))\n",
    "\n",
    "# 绘制图形\n",
    "markers = {'train': 'o', 'test': 's'}\n",
    "x = np.arange(len(train_acc_list))\n",
    "plt.plot(x, train_acc_list, label='train acc')\n",
    "plt.plot(x, test_acc_list, label='test acc', linestyle='--')\n",
    "plt.xlabel(\"epochs\")\n",
    "plt.ylabel(\"accuracy\")\n",
    "plt.ylim(0, 1.0)\n",
    "plt.legend(loc='lower right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的例子中，每经过一个 epoch，就对所有的训练数据和测试数据计算识别精度，并记录结果。之所以要计算每一个 epoch 的识别精度，是因为如果在for语句的循环中一直计算识别精度，会花费太多时间。并且也没有必要那么频繁地记录识别精度（只要从大体方向上大致把握识别精度的推移就可以了）。因此，我们才会每经过一个 epoch 就记录一次训练数据的识别精度。\n",
    "\n",
    "上图中，实现表示训练数据的识别精度，虚线表示测试数据的识别精度。如图所示，随着学习的进行，我们发现使用训练数据和测试数据评价的识别精度都提高了，并且，这两个识别精度基本上没有差异（两条线重叠在一起）。因此，可以说这次的学习中没有发生过拟合的现象。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 小节\n",
    "\n",
    "我们学习了神经网络的学习。首先，为了能顺利进行神经网络的学习，我们导入了损失函数这个指标。以这个损失函数为基准，找出使它的值达到最小的权重参数，就是神经网络的学习目标。为了找到尽可能小的损失函数值，我们介绍了使用函数斜率的梯度法。\n",
    "\n",
    "* 机器学习中使用的数据集分为训练数据和测试数据。\n",
    "* 神经网络用训练数据进行学习，并用测试数据评价学习到的模型的泛化能力。\n",
    "* 神经网络的学习以损失函数为指标，更新权重参数，以使损失函数的值减小。\n",
    "* 利用某个给定的微小值的差分求倒数的过程，就是数值微分。\n",
    "* 利用数值微分，可以计算权重参数的梯度。\n",
    "* 数值微分虽然费时间，但是实现起来很简单。下一章中要实现的稍微复杂一些的误差反向传播法可以高速地计算梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
